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CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  year  1979  was  marked 
by  the  invasion  of  Afghanistan 
by  the  Soviet  Union,  the 
Iranian  Revolution  that  over¬ 
threw  the  Shah  and  held  52 
American  diplomats  hostage, 
and  the  election  of  Margaret 
Thatcher  as  Prime  Minister  of 
the  United  Kingdom.  The  Sony 
Walkman  was  introduced  to  a 
transfixed  public,  but  at  a  price 
equal  to  almost  a  week's  wages.  The  entire  federal  budget  was 
roughly  $500  billion,  a  number  now  equated  with  just  the 
Department  of  Defense  portion  of  the  budget. 

The  year  was  also  marked  by  the  first  full  publication  of 
the  now-legendary  “Augustine's  Laws  and  Major  System 
Development  Programs''  in  the  pages  of  the  Defense  Systems 
Management  Review,  and  a  few  years  later  in  Concepts,  the 
journals  that  preceded  the  current  Defense  Acquisition 
Research  Journal.  Norm  Augustine,  at  the  time  a  member  of 
the  journal's  Editorial  Board  and  a  Vice-President  at  Martin 
Marietta  Aerospace,  was  well  known  in  the  defense  acquisi¬ 
tion  community,  having  served  in  several  senior  leadership 
positions  in  the  Department  of  Defense  and  industry. 

Although  Augustine's  Laws  were  written  in  a  somewhat 
tongue-in-cheek  fashion,  they  are  quite  serious.  We  are 
proud  to  republish  them  35  years  later,  and  delighted  that  he 
has  agreed  to  pen  a  short  introduction  looking  back  at  how 
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they  have  held  up  over  time.  His  most  famous  law,  on  the 
increasing  cost  of  tactical  aircraft,  can  now  fix  the  precise 
date  on  which  the  entire  defense  budget  will  buy  just  one 
airplane.  We  are  even  provided  a  glimpse  into  future  laws. 

In  this  issue  we  are  also  pleased  to  publish  the  research 
of  several  teams  of  researchers  who  are  extending  the 
boundaries  of  our  knowledge  in  defense  acquisition.  Alex 
Miller  and  Joshua  L.  Ray,  in  “Moving  from  Best  Practices 
to  Standard  Practices  in  Defense  Acquisition,”  examine 
the  DoD's  difficulty  in  translating  observed  best  practices 
into  consistently  applied  acquisition  processes  and  provide 
a  simple  framework  to  assess  the  standardization  of  these 
practices.  Capt  Allen  J.  DeNeve,  USAF,  Lt  Col  Erin  T.  Ryan, 
USAF,  Lt  Col  Jonathan  D.  Ritschel,  USAF,  and  Christine 
Schubert  Kabban,  in  “Taming  the  Hurricane  of  Acquisition 
Cost  Growth— Or  at  Least  Predicting  It,”  provide  a  statisti¬ 
cally  derived  approach  to  forecast  how  a  program's  baseline 
is  likely  to  change  over  time,  instead  of  assuming  it  will 
remain  static,  which  promises  to  improve  the  prediction 
of  a  program's  likely  cost  growth  and  thus  to  develop  more 
realistic  cost  estimates. 

In  “The  Effects  of  System  Prototype  Demonstrations  on 
Weapon  Systems  Development,"  Edward  J.  Copeland,  Thomas 
H.  Holzer,  Timothy  J.  Eveleigh,  and  Shahryar  Sarkani  find 
that  system  prototyping  positively  influences  the  outcomes 
of  weapon  systems  development  programs— a  key  finding 
that  helps  support  recent  Better  Buying  Power  initiatives 
promulgated  by  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  Frank  Kendall  to  incentivize  inno¬ 
vation  in  both  government  and  industry. 

Finally,  we  acknowledge  the  hard  work  and  dedication  of 
three  departing  members  ofth e  Defense  Acquisition  Research 
Journal's  Editorial  Board:  Dr.  J.  Robert  Wirthlin  of  the  Air 
Force  Institute  of  Technology,  Dr.  Donald  Hutto  of  the  Defense 
Acquisition  University,  and  Dr.  Mark  Montroll  of  the  Dwight 
D.  Eisenhower  School  for  National  Security  and  Resource 
Strategy.  We  also  welcome  three  new  members  to  the  Board: 
Dr.  John  M.  Colombi  of  the  Air  Force  Institute  of  Technology, 
and  Professors  John  Cannady  and  Dana  Stewart,  both  of  the 
Defense  Acquisition  University  South  Region  in  Huntsville, 
Alabama.  We  look  forward  to  their  contributions. 
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DAU  CENTER 
FOR  DEFENSE 
ACQUISITION 
RESEARCH 

RESEARCH  AGENDA  2015 


The  Defense  Acquisition  Research  Agenda  is  intended 
to  make  researchers  aware  of  the  topics  that  are,  or  should 
be,  of  particular  concern  to  the  broader  defense  acquisition 
community  throughout  the  government,  academic,  and 
industrial  sectors.  The  purpose  of  conducting  research  in 
these  areas  is  to  provide  solid,  empirically  based  findings  to 
create  abroad  body  of  knowledge  that  can  inform  the  devel¬ 
opment  of  policies,  procedures,  and  processes  in  defense 
acquisition,  and  to  help  shape  the  thought  leadership  for  the 
acquisition  community. 

Each  issue  of  the  Defense  ARJ  will  include  a  different  selec¬ 
tion  of  research  topics  from  the  overall  agenda,  which  is  at: 
http://www.dau.mil/research/Pages/researchareas.aspx 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to 
measure  the  effect  on  defense  acquisition  costs 
of  maintaining  an  industrial  base  in  various 
sectors? 

•  What  means  exist  (or  can  be  developed)  of  mea¬ 
suring  the  effect  of  utilizing  defense  industrial 
infrastructure  for  commercial  manufacture 
in  growth  industries?  In  other  words,  can  we 
measure  the  effect  of  using  defense  manufac¬ 
turing  to  expand  the  buyer  base? 
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•  What  means  exist  (or  can  be  developed)  to 
determine  the  degree  of  openness  that  exists 
in  competitive  awards? 

•  What  are  the  different  effects  of  the  two  best- 
value  source-selection  processes  (tradeoff  vs. 
lowest  price  technically  acceptable)  on  pro¬ 
gram  cost,  schedule,  and  performance? 

Strategic  Competition 

•  Is  there  evidence  that  competition  between 
system  portfolios  is  an  effective  means  of  con¬ 
trolling  price  and  costs? 

•  Does  lack  of  competition  automatically  mean 
higher  prices?  For  example,  is  there  evidence 
that  sole  source  can  result  in  lower  overall 
administrative  costs  at  both  the  government 
and  industry  levels,  to  the  effect  of  lowering 
total  costs? 

•  What  are  the  long-term  historical  trends  for 
competition  guidance  and  practice  in  defense 
acquisition  policies  and  practices? 

•  To  what  extent  are  contracts  being  awarded 
noncompetitively  by  congressional  mandate, 
for  policy  interest  reasons?  What  is  the  effect 
on  contract  price  and  performance? 

•  What  means  are  there  (or  can  be  developed) 
to  determine  the  degree  to  which  competitive 
program  costs  are  negatively  affected  by  laws 
and  regulations  such  as  the  Berry  Amendment 
and  Buy  American  Act? 
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Preface: 


Augustine's  Laws 
and  Major  System  * 

DEVELOPMENT  PROGRAMS  <41 

by  Norman  R.  Augustine 


Two  score  and  eight  years  ago  (somehow  that  doesn't  sound  as 
long  as  48  years  ago),  I  was  working  as  Assistant  Director  of  Defense 
Research  and  Engineering  in  the  Office  of  the  Secretary  of  Defense, 
and  it  occurred  to  me  that  just  as  physical  systems  obey  certain  laws  of 
nature,  perhaps  defense  acquisition  obeys  certain  laws  of  human  nature. 
To  my  amazement— and  everlasting  regret— this  turned  out  to  be  true. 
(Caveat:  To  protect  the  innocent  as  well  as  the  guilty,  the  views  expressed 
in  Augustine's  Laws  are  purely  my  own  and  do  not  necessarily  reflect  the 
views  of  my  employers— past,  present,  or  future.) 

The  earliest  of  my  immutable  laws— perhaps  the  most  infamous  among 
them— addressed  the  increasing  cost  of  tactical  aircraft.  It  showed 
that  the  unit  cost  of  such  machines  increases  at  a  very  predictable 
rate— a  factor  of  four  every  10  years  (6  db/decade  for  the  electrical  engi¬ 
neers)— independent  of  everything  else  (performance,  quantity,  military 
department,  inflation,  etc.).  This  led  to  the  following  law,  based  on  a 
straightforward  extrapolation  of  the  defense  budget  and  the  entire  half- 
century's  experience  then  available  in  building  military  aircraft: 


Recognizing  that  this  represents  a  not  inconsiderable  extrapolation,  I 
rationalized  that  economists  in  Washington  frequently  extrapolate 
based  on  a  single  data  point. 


In  the  year  2054,  the  entire  defense  budget  will  buy  just 
one  tactical  aircraft...which  will  have  to  be  shared  by 
the  Navy  and  the  Air  Force  6  months  each  year,  with 
the  Marine  Corps  borrowing  it  on  the  extra  day  during 
leap  years. 
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Fast-forwarding  to  today,  The  Economist 
magazine  recently  devoted  a  full  page  to 
updating  this  law,  which  was  initially  pro¬ 
mulgated  in  1967  (and  published  35  years  ago 
in  the  Defense  Systems  Management  Review). 
The  Economist's  analysis  confirmed  that  the 
prediction  is  still  right  on  track.  In  fact,  it  is  now 
possible  to  refine  the  previously  projected  2054  date: 
it  will  actually  be  July  23, 2054.  Indeed,  the  number  of  aircraft  being 
built  each  year  has  declined  precipitously  (96,000  per  year  at  the  peak 
of  World  War  II)  and,  as  predicted  by  Law  XI V— using  Roman  numerals 
makes  it  seem  more  profound— the  pilots  of  said  aircraft  are  gradually 
being  squeezed  right  out  of  the  cockpit. 


It  is  therefore  with  considerable  regret  that  one  must  conclude  that  most 
of  my  laws  have  in  fact  withstood  the  test  of  time.  The  principal  excep¬ 
tion  wherein  they  seem  to  have  missed  the  mark  is  my  projection  of  the 
ineffectiveness  of  Congress  in  carrying  out  its  principal  role  in  defense 
acquisition— an  attainment  that  has  considerably  underperformed  my 
lack  of  expectations. 


Consider  Congress's  major  responsibility  to  produce  a  budget  for 
national  defense  as  assigned  in  the  Constitution.  In  my  initial  set  of 
laws  published  in  Defense  Systems  Management  Review,  it  was  noted 
that  Congress  was  slipping  later  and  later  into  each  successive  fiscal  year 
before  it  produced  an  operating  budget  (note  that  in  contrast  to  virtually 
any  commercial  firm,  Congress  does  not  even  attempt  to  produce  a  capi¬ 
tal  budget).  Finally,  the  point  was  reached  in  the  mid-1970s  where  fully 
60  percent  of  a  year  had  passed  before  a  budget  was  provided. 


Unabashedly  extrapolating  the  above  trend,  I  was  able  to  predict  that  in 
another  13  years  from  that  time  no  budget  would  be  provided  until  the 
fiscal  year  had  passed  into  history.  As  a  Russian  acquaintance  explained 
to  me  at  the  end  of  the  Cold  War,  speaking  in  his  case  of  his  nation's  pro¬ 
pensity  for  historical  revisionism,  “Not  only  do  we  have  an  uncertain 
future,  we  have  an  uncertain  past." 

Drawing— incorrectly  as  it  turned  out— on  my  experience  in  industry, 
I  concluded  that  this  level  of  nonperformance  would  prompt 
an  immediate  uprising  on  Capitol  Hill  that  would  focus  on 
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discipline,  accountability,  and  consequences  throughout  the  legislative 
process.  But  as  it  turned  out,  the  legislative  branch  had  a  far  more  imagi¬ 
native  solution  to  the  dilemma  than  merely  implementing  the  principles 
of  Management  101.  Instead,  in  1976  Congress  simply  redefined  the  fis¬ 
cal  year,  slipping  it  by  3  months...thus  (presumably),  making  it  possible 
to  produce  a  budget  on  time  once  again  (overlooking  the  minor  arith¬ 
metical  inconsistency  inherent  in  this  illogic). 


On  the  other  hand,  Congress 
has  done  a  rather  good  job  of 
placing  demands  on  others.  For  example,  over  the  most  recent  one-third 
of  a  century  the  number  of  reports  it  requires  the  Executive  Branch  to 
submit  by  a  certain  date  has  increased  by  no  less  than  351  percent.  And 
in  its  newly  available  free  time  the  Congress  has  increased  the  income 
tax  code  from  a  mere  16  pages  80  years  ago  to  45,622  pages  today— while 
legislating  that  ignorance  of  the  law  is  no  excuse. 


27 


The  problem,  of  course,  was  that  the  date 
of  the  National  Defense  Appropriations 
Act  immediately  began  slipping  further 
and  further  into  the  new  fiscal  year  just 
as  it  had  in  the  old  fiscal  year,  until  in  the 
most  recent  5-year  period  the  year  was  38 
percent  over,  on  average,  before  a  budget  was 
approved.  Presumably,  another  redefinition 
^  4  of  the  fiscal  year  will  be  forthcoming, 
followed  by  yet  another,  thus  provid¬ 
ing  a  never-ending  solution  to  the 
tardy-budget  dilemma— a  sort  of 
self-eating  watermelon. 


The  original  bookform  of  Augustine's  Laws,  now  published  in  six  lan¬ 
guages,  contained  52  laws.  But,  it  has  turned  out  that  creating  laws  is 
such  a  target-rich  battlefield  that  I  have  now  collected  more  than  enough 
material  to  proclaim  yet  another  52  laws— but,  probably  fortunately,  with 
no  time  to  record  them. 


Many  of  the  original  laws  about  defense  acquisition  have  actually  been 
found  to  have  application  to  a  wide  range  of  fields,  spanning  from  health¬ 
care  to  education  and  well  beyond.  Consider,  for  example,  the  difficulty  of 
producing  more  engineers— a  profession  critical  to  the  defense  acquisi¬ 
tion  process.  Of  the  93  nations  evaluated  in  one  recent  study,  the  fraction 
of  baccalaureate  degrees  going  to  engineers  placed  the  United  States 
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solidly  in  79th  place  (most  closely  matching  Mozambique).  Worse  yet,  the 
National  Assessment  of  Educational  Progress  (NAEP)  standardized  test 
scores  in  mathematics  for  U.S.  high  school  students  have  not  improved 
in  the  past  half-century,  even  though  public  school  spending  per  student 
has  grown  markedly.  Thus,  in  1970  it  cost  just  $15.30  per  student  per 
point  scored  on  the  NAEP  mathematics  test,  whereas  today  the  cost 
is  $36.07  (in  constant  dollars).  This  does  not  seem  to  bode  well  for  the 
future  of  defense  acquisition... or,  for  that  matter,  the  nation's  economy. 
One  creative  solution  would,  of  course,  be  to  increase  the  maximum 
number  of  points  that  could  be  scored  on  the  examination. 


When  I  first  began  publicly  proclaiming  laws  about  the  failures  of  indus¬ 
try  and  government  it,  perhaps  not  unreasonably,  stirred  a  degree  of 
angst  on  the  part  of  my  employer.  I  was  reminded  that,  being  a  defense 
contractor,  people  who  live  in  glass  houses  should  not  throw  stones... 
or  something  to  that  effect.  However,  other  than  one  minor  episode— a 
friend  of  mine,  then  Chief  of  Staff  of  the  Army,  took  public  umbrage  at 
the  law  I  had  endorsed,  which  states  that  “Rank  times  IQ  is  a  constant” 
(sadly,  this  applies  in  industry,  too)— the  laws  have  generally 
been  embraced  as  merely  unwelcome  nuisances. 

I  actually  received  a  note  from  Yogi  Berra  saying,  “If  you'll 
promise  to  stay  out  of  the  laws  business,  I'll  promise 
to  stay  out  of  the  airplane  business!''  But 
much  more  condescending  was  the  let¬ 
ter  I  received  from  Laurence  Peter,  of 
the  Peter  Principle,  asserting  that  I 
had  undermined  his  entire  life's  work. 

He  said  that  I  had  risen  not  one,  but 
two  levels  above  my  level  of  compe¬ 
tence.  This  hazard  of  proclaiming 
new  laws  was  perhaps  best 
described  in  a  three-sentence 
essay  about  Socrates  writ¬ 
ten  by  a  fourth-grader: 

“Socrates  was  a  philos¬ 
opher,''  she  wrote.  “He 
went  around  telling 
people  what  was  wrong. 

They  fed  him  hemlock.'' 
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Whatever  the  case,  the  gauntlet  laid  down  by  Augustine's  Laws  seems  to 
be  more  relevant  today— and  certainly  more  demanding— than  was  the 
case  at  the  time  they  were  conceived.  And,  for  the  record,  my  newest  law 
goes  as  follows: 

If  you  send  money  to  the  management  of  a  project  that 
is  in  trouble,  they  will  remember  you  the  next  time  they 
need  money. 

You  first  read  it  here. 


Norman  R.  Augustine 

Mr.  Augustine  is  retired  Chairman  and  CEO 
of  Lockheed  Martin  and  served  as  Under 
Secretary  of  the  Army,  Chairman  of  the 
American  Red  Cross,  President  of  the  Boy 
Scouts,  Chairman  of  National  Academy  of 
Engineering,  Defense  Science  Board,  and 
American  Institute  of  Aeronautics  and 
Astronautics.  He  is  a  holder  of  National 
Medal  of  Technology  and  five-time 
recipient  of  the  Department  of  Defense 
Civilian  Distinguished  Service  Medal. 
Mr.  Augustine  is  the  recipient  of  33  honorary  doctorates  and  served 
as  Trustee  of  Princeton,  MIT,  and  Johns  Hopkins  and  Regent  of 
University  System  of  Maryland. 


For  screen-reader  access:  Please  click  here  for  a  text  alternative. 
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Augustine’s  Laws 
and  Major  System 
Development  Programs 

Norman  R .  Augustine 

I  nsight  into  the  problems  of  program  management  is  sometimes  found  in 
unexpected  places.  For  example.  A,  A,  Milne  could  well  have  been  writing  about 
the  sufferings  of  managers  of  large  system  development  activities  in  the  opening 
paragraph  of  Winnie-the-Pooh,  "Here  is  Edward  Bear/'  he  wrote,  "coming 
downstairs  now,  bump,  bump,  bump,  on  the  back  of  his  head,  behind 
Christopher  Robin.  It  is,  as  far  as  he  knows,  the  only  way  of  coming  downstairs, 
but  sometimes  he  feels  that  there  really  is  another  way... if  only  he  could  stop 
bumping  for  a  moment  and  think  of  it!" 

Indeed,  there  is  a  better  way,  as  innumerable  highly  successful  programs  have 
demonstrated.  Still,  there  remains  that  large  set  of  much  maligned  projects 
which,  were  they  ever  to  be  documented  into  a  movie,  might  best  be  viewed  with 
the  film  run  backward  in  order  to  ensure  a  happy  ending.  It  is  largely  from  these 
programs  that  Augustine's  Laws  have  been  formulated.  The  laws  are  dedicated  to 
the  proposition  that  with  a  better  understanding  of  the  history  of  past  programs, 
one  need  only  selectively  repeat  history  in  the  future.  Further,  it  is  suggested  that 
the  behavior  of  large  system  management  activities  is  as  amenable  to  analysis  as 
are  most  of  the  systems  themselves.  Each  of  the  15  laws,  with  a  sample  of  the 
evidence  supporting  its  existence,  is  examined  in  the  following  paragraphs. 

Employer  of  Only  Resort 

Law  Number  1  corroborates  the  late  Senator  Everett  Dirksen's  observation 
about  big  government:  "A  billion  here,  and  a  billion  there/'  he  stated,  "and  pretty 
soon  it  adds  up  to  real  money." 

The  percentage  of  civilian  workers  in  the  United  States  employed  by  govern¬ 
ment  at  the  Federal,  State  and  local  levels  is  displayed  in  Figure  1.  A  growth  trend 
is  observed  which  has  been  very  predictable  and  monotonic  throughout  the 
history  of  the  nation,  A  modest  extrapolation  into  the  future,  shown  by  the  dot¬ 
ted  portion  of  the  trend  line,  indicates  that  the  time  is  not  too  distant  when  one 


Authors  Note;  "Augustine's  Laws"  are  intended  to  help  explain  the  tribulations  of  program 
management .  They  have  been  formulated  over  a  period  of  years  and  are  based  on  observations  of  a 
large  number  of  actual  development  programs.  Although  some  of  the  laws  have  b«n  published 
previously,  this  U  the  first  time  that  all  IS  laws  have  been  collected  as  an  entity. 


Norman  R.  Augustine  is  Vice  fVestdenf  of  Martin  Marietta  Aerospace  responsible  for  research, 
development  and  manufacturing.  He  has  served  as  Under  Secretary  of  the  Army ,  Assistant  Secretary 
of  the  Army  for  Research  and  Development,  and  as  Assistant  Director  of  Defense  Research  and 
Engineering  hi  the  Office  of  the  Secretary  of  Defense.  His  recognition  in  1076  of  the  need  for  a  publko- 
ticm  devoted  to  the  concerns  of  those  involved  in  acqidsttkm  management  resulted  in  the  establish* 
ment  of  the  Defense  Systems  Management  Review.  Hi  continues  to  serve  the  Review  as  a  member  of 
the  Editorial  Board.  Mr.  Augustine  holds  B.  S.  E.  and  M.5.  E  degree*,  in  aeronautical  engineering  from 
Princeton  University. 
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should  expect  100  percent  of  the  working  population  to  be  working  for  the 
government.  Taking  the  next  logical  step,  one  can  then  state  Augustine's  First 
Law: 

I:  By  the  time  of  the  nation's  IricentermiaL  there  will  he  more  government 
workers  in  the  United  States  than  there  are  workers. 

For  Wfcaf  It's  Worth,  Save  Your  Money 

This  trend  in  the  growth  of  government  as  measured  by  the  number  of  people 
it  employs  is,  of  course,  paralleled  by  the  government's  financial  receipts;  and  in 
turn  by  the  government's  ability  even  to  conduct  its  own  programs  on  its  own 
behalf  as  it  sees  fit.  For  example,  there  is  now  a  tax  collector  somewhere  in  the 
U.S,  extracting  a  dollar  every  25  milliseconds — including  roughly  half  of  each 
dollar  of  the  profits  earned  by  industry.  By  extrapolating  the  trend  shown  in 
Figure  2,  it  can  be  seen  that  the  government  will  have  all  the  money  that  is 
generated  in  the  ITS-  economy  by  the  year  2120  and,  as  has  already  been  noted,  it 
will  directly  employ  all  the  people  about  60  years  prior  to  that  time.  What  hap¬ 
pens  during  the  interim  period  between  these  dates  is  not  yet  clear,  but  poses  the 
interesting  question  of  whether  the  last  person  left  in  the  private  sector  will  have 
to  support  the  entire  nations  work  force,  or  whether  he  or  she  instead  will  in¬ 
dividually  enjoy  the  full  benefit  of  those  residual  funds  not  yet  controlled  by  the 


Defense  ARJ,  January  2015,  Vol.  22 No.  1:2-63 


9 


FIGURE  1 

The  Demise  of  Free  Enterprise 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


52  II  Defense  Systems  Management  Review 


10 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


Augustine fs  Laws  |  53 


government,  Whatever  the  explanation,  this  uncertainty  leads  to  the  guarded  op¬ 
timism  expressed  in  Law  Number  IL  actually  the  corollary  to  the  first  law,  which 

fat 

U:  People  working  in  the  private  sector  should  try  to  save  money  if  at  all  feasi¬ 
ble.  There  remains  a  possibility  that  it  may  someday  be  valuable  again. 

In  terms  of  the  fraction  of  the  gross  national  product  absorbed  in  the  form  of 
government  receipts,  one  can  also  use  the  extrapolation  presented  in  figure  2  to 
ascertain  that  the  ILS,  lags  England  by  only  17  years  and  Sweden  by  only  56 
years  in  this  respect. 

The  significance  of  these  observations  to  an  industrial  program  manager  is  ob¬ 
vious.  Their  significance  to  a  government  program  manager,  although  perhaps 
less  obvious,  is  nonetheless  every  bit  as  significant;  namely,  competition  among 
potential  sources  is  the  essence  of  a  program  manager's  leverage,  and  the  absence 
of  a  multiplicity  of  strong  competitors  can  only  lessen  the  government  program 
manager's  chances  of  success. 

On  Striving  to  Be  Average 

Law  Number  III  confirms  the  suspicion  that  very  few  people  come  up  to  the 
average. 

The  contribution  made  by  a  group  of  people  working  in  a  common  endeavor 
tends  to  be  highly  concentrated  in  the  achievements  of  a  few  members  of  that 
group.  The  degree  of  this  concentration  is  observed  to  obey  a  fundamental  law, 
as  indicated  by  the  data  in  Figure  3,  It  is  seen  that  the  great  predominance  of  out¬ 
put  is  produced  by  a  disproportionately  small  segment  of  the  participants,  with 
the  same  law  seeming  to  apply  whether  one  is  addressing  authors,  pilots, 
engineers,  policemen,  or  football  players.  As  one  "digs  deeper  into  the  barrel,"  so 
to  speak,  in  order  to  increase  the  manning  of  a  given  task,  the  average  output  is 
merely  driven  downward  and,  ultimately,  large  numbers  of  participants  are 
added  with  hardly  any  increase  in  productivity  at  all  (unless,  of  course,  changes 
in  work  methods  are  also  introduced).  Conversely,  substantial  reductions  in 
manning— eliminating  the  least  productive  contributors— can  be  made  with  little 
impact  on  overall  output.  In  fact,  the  least  productive  half  of  all  participants 
seems  to  generate  no  more  than  20  percent  of  the  total  output. 

It  might  be  more  accurate  to  describe  the  above  observation  as  merely  a 
generalization  or  corollary  of  V.  Pareto's  work  published  in  1897,  in  which  it  was 
demonstrated  that  the  proportion  of  people  with  an  income  N  was  proportional 
to  1/1 V-s* 

The  results  presented  in  Figure  3  are  probably  understated,  since  the  data  base 
considers  only  participants  who  made  at  least  some  contribution,  such  as  obtain¬ 
ing  one  patent,  when  in  reality  there  are  many  who  obtained  no  patents.  Further, 
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FIGURE  3 

Concentration  of  Productivity 
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there  are  unquestionably  those  who  produce  negative  output,  such  as  the  worker 
who  makes  so  many  mistakes  that  a  great  deal  of  the  time  of  other  potentially 
productive  workers  is  consumed  in  rectifying  the  problems  the  former  has 
created.  Only  about  one-third  of  the  workers  typically  achieve  a  level  of  con¬ 
tribution  equal  to  the  average  of  all  those  who  contribute. 

This  leads  to  the  third  law,  which  relates  to  the  allocation  of  manpower  and 
can  be  stated  as  follows : 

III:  One- tenth  of  the  people  involved  in  a  given  endeavor  produce  at  least  one- 
third  of  the  output,  increasing  the  number  of  participants  merely  serves 
to  reduce  the  average  performance. 

As  has  often  been  pointed  out.  when  an  individual  item  can  only  be  produced 
at  a  financial  loss,  it  is  very,  veiy  difficult  to  make  it  up  on  volume. 
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FIGURE  4 

Accuracy  of  Projecting  Accomplishment  Date  for  Major  Milestones 
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The  Reality  of  the  Fantasy  Factor 

Law  Number  IV  explains  why  one  should  never  commit  to  complete  a  task 
within  6  months  of  the  end  of  any  fiscal  year — in  either  direction. 

In  1798,  Eli  Whitney  contracted  to  deliver  10,000  muskets  to  the  Continental 
Army  within  28  months.  As  things  worked  out,  he  delivered  them  in  37  months, 
or  in  about  one-third  more  time  than  had  been  anticipated. 

During  1978,  a  number  of  new  systems  were  delivered  to  the  ILS.  military 
forces  by  major  defense  contractors.  On  the  average,  according  to  the  reports 
submitted  to  the  Congress,  these  systems  were  delivered  in  about  one-third  more 
time  than  had  been  anticipated. 

The  fraction  "'one-third”  seems  to  have  scientific  significance  in  determining 
the  schedule  error  associated  with  predicting  major  program  events  (some  say  the 
correct  number  is  actually  more  nearly  equal  to  one  over  pi).  The  data  shown  In 
Figure  4  are  derived  from  a  large  number  of  official  schedule  estimates  predicting 
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when  various  milestones,  such  as  first  flight,  first  delivery,  etc.,  will  occur.  These 
data,  in  turn,  form  the  basis  for  Law  Number  IV,  which  defines  the  concept  of  the 
Fantasy  Factor; 

IV:  Any  given  task  can  be  completed  In  only  one-third  more  time  than  is  cur¬ 
rently  estimated + 

The  above  law  addresses  the  accuracy  of  predicting  how  long  it  will  take  to 
reach  any  particular  milestone  in  a  development  program's  life.  A  different  law 
addresses  the  overall  trend  of  increasing  time  required  actually  to  prosecute  a 
development  program.  This  latter  issue  is  the  subject  of  Law  Number  V. 

So  Old  for  Its  Age 

Law  Number  V,  on  program  geriatrics,  explains  how  World  War  II  was  won 
in  about  half  the  time  it  today  takes  to  develop  a  new  military  system. 

Figure  5  shows  that  the  average  major  system  development  for  national 
defense  today  takes  slightly  over  8  years  to  complete.  Interestingly,  the  doing 
time,  for  example  the  time  from  the  beginning  of  the  design  of  a  new  airplane 
until  its  first  flight,  has  not  changed  significantly  during  the  last  quarter  of  a  cen¬ 
tury,  as  can  be  seen  in  Figure  6, 


FIGURE  5 

Duration  of  Development  Programs 
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What  has  changed  is  the  decision/ approve  I  time  it  takes  to  get  a  new  program 
started,  together  with  the  time  it  takes  to  get  it  fielded  once  the  development  has 
been  completed.  The  historical  ratio  of  planning  time  to  doing  time  lor  a  number 
of  major  system  developments  is  shown  in  Figure  7,  On  the  average.,  the  total 
time  it  takes  to  develop  a  new  system F  including  decision  and  approval  time,  has 
been  increasing  at  a  rate  of  3  months  per  year,  each  year,  for  the  past  15  years. 

La  w  Number  V  can  then  be  stated,  based  in  part  on  the  fact  that  the  half-life 
of  most  technologies  has  been  determined  elsewhere  to  be  on  the  order  of  10 
years: 

V:  If  current  trends  persist,  most  new  systems  wilt  be  obsolete  only  slightly 
before  they  are  bom. 

Work  and  the  Theory  of  Relativity 

Law  Number  VI  offers  an  alternative  to  the  bus  company  serving  the  Bagnail 
to  Greenfields  route  in  England,  whose  spokesman  recently  countered  criticism 
that  half-empty  buses  were  speeding  past  long  queues  of  would-be  riders  with  the 
explanation,  "It  is  impossible  for  the  drivers  to  keep  their  timetable  if  they  have  to 
stop  for  passengers," 

In  competitive,  time-sensitive  markets,  managers  are  simultaneously 
challenged  on  three  fronts.  Not  only  must  they  produce  a  desirable  product  at  a 
reasonable  price,  but  in  addition  they  must  deliver  their  output  to  the 
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FIGURE  7 
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marketplace  in  a  timely  manner.  This  urgency  is  characteristic  of  a  large  variety 
of  products,  irrespective  of  whether  the  aforementioned  pressure  arises  from 
perishability  of  the  product  itself,  the  need  to  rapidly  exploit  some  technological 
breakthrough,  or  merely  to  keep  up  with  demand. 

In  environments  wherein  only  one  source  of  an  item  1$  available,  however,  an 
altogether  different  set  of  dynamics  prevails.  Consider,  as  but  one  example,  the 
problem  faced  by  the  U.S.  Congress  as  it  each  year,  in  addition  to  a  myriad  of 
other  crucial  tasks,  pursues  the  matter  of  approving  a  budget  for  each  of  the 
Federal  departments.  For  one  reason  or  another,  the  Congress  has  apparently 
found  it  increasingly  difficult  to  complete  this  task  prior  to  the  beginning  of  the 
year  in  which  the  money  is  to  be  spent . 

The  data  in  Figure  8  display  how  in  each  fiscal  year  the  date  at  which  funds 
are  finally  appropriated  has  tended  to  slide  further  and  further  into  the  year.  This 
problem  recently  culminated,  in  the  case  of  the  defense  budget,  in  a  circumstance 
wherein  the  appropriation  act  did  not  become  law  until  the  year  was  more  than 
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FIGURE  G 

Increasing  Length  of  Budget  Approval  Process 
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half  completed]  The  challenge  posed  to  those  charged  with  executing  that  budget 
can,  in  fact,  be  accurately  imagined,  r„  particularly  those  unfortunate  managers 
whose  requested  budget  was  halved. 

What  the  Future  portended  for  those  same  managers  could  be  glimpsed  by 
projecting  forward  in  time  the  trend  line  in  Figure  5.  The  inevitable  conclusion 
seemed  to  be  that  It  would  be  only  about  a  decade  until  the  situation  reached 
crisis  proportions;  i.e,„  the  budget  would  not  be  approved  until  the  year  was 
altogether  past. 

Fully  recognizing  this  dilemma,  the  Congress  proceeded  to  rectify  the  intensi¬ 
fying  problem  with  both  alacrity  and  decisiveness.  Less  imaginative  managers  in 
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private  industry,  given  the  same  circumstances,  might  have  resorted  to  such  con¬ 
ventional  techniques  as  eliminating  some  oF  the  18  votes  taken  each  year  on  large 
segments  of  the  budget,  or  even  to  a  process  of  expediting  the  budget  cycle  by 
combining  various  steps  in  the  review  process,  or  perhaps  even  by  resorting  to 
multi-year  funding. 

As  luck  would  have  it,  however,  no  such  pedestrian  approaches  were  needed. 
The  obvious  solution,  and  that  seized  upon  by  the  Congress,  was,  of  course,  to 
pass  a  law  changing  the  definition  of  the  fiscal  year,  hereafter  slipping  it  neatly  in¬ 
to  compliance  with  the  time  it  was  actually  taking  to  complete  the  task  of  prepar¬ 
ing  a  budget. 

The  essential  element  that  made  this  resolution  of  a  nasty  problem  possible 
was,  of  course,  the  fact  that  there  is  only  one  Congress  available,  and  if  this  one 
does  not  produce  a  budget  act  by  any  given  time,  there  is  no  danger  of  another 
competitive  Congress  stepping  in  and  producing  one  of  its  own.  It  can  be  safely 
inferred  that  such  latitude  for  problem  solving  is  by  no  means  restricted  to 
governmental  bodies,  but  is  attendant  to  any  entity  functioning  in  a  sole-source 
environment. 

Professor  C,  Northcote  Parkinson,  in  the  well-known  law  which  bears  his 
name,  examined  the  effort  devoted  to  activities  which  are  time-constrained.  Law 
Number  VI  of  the  present  monograph  is  a  reciprocal  to  Parkinson's  proposition, 
and  considers  the  case  wherein  the  work  to  be  performed  is  constrained.  Parkin¬ 
son's  Law  pointed  out,  in  essence,  that  work  expands  to  fit  the  time  prescribed.  In 
contradistinction: 

VI:  In  a  noncompetitive  process,  time  expands  to  fit  the  work  prescribed. 

The  Impossible  Only  Takes  a  Little  Longer 

Law  Number  VII  explains  why  one  professional  football  coach,  after  having 
been  given  an  unlimited  budget  by  the  club's  owner,  was  accused  before  the 
season  had  begun  of  having  overspent  it. 

Two  types  of  uncertainty  plague  most  major  programs:  known-unknowns 
and  unknown-unknowns.  The  known-unknowns,  such  as  the  composition  of  the 
moon's  surface  at  the  exact  location  of  the  first  Apollo  landing,  can  be  accom¬ 
modated  and  a  program  planned  which  hedges  against  their  consequences.  The 
second  category,  the  unknown-unknowns,  cannot  be  specifically  identified  in  ad¬ 
vance,  but  their  existence  can  be  predicted  with  every  bit  as  much  confidence  as 
insurance  companies  place  in  actuarial  statistics.  An  example  of  the  latter 
category  of  unknown  is  the  lightning  that  struck  Apollo  XII  shortly  after  its 
launch  on  the  way  to  the  moon. 

Somehow,  in  every  major  program,  "  lightning''  strikes  someth here.  It  cannot 
be  predicted  where  it  will  strike,  only  that  it  will  strike  But,  unfortunately,  the 
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budgeting  system  used  in  defense  planning  has  not  at  least  until  recently,  permit¬ 
ted  the  recognition  of  such  contingencies  or  the  provision  of  lightning  rods.  This 
was  in  part  due  to  the  vulnerability  of  so-called  management  reserves  to  congres¬ 
sional  budget  cutting,  and  partly  due  to  optimistic  bids  engendered  in  a  cost- 
reimbursable  competitive  contract  award  environment. 

Although  there  are  available  many  more  sophisticated  ways  of  predicting  pro¬ 
gram  costs  were  one  in  fact  to  use  them,  the  cost-estimating  correction  factor 
presented  in  Figure  9  would,  in  the  aggregate,  have  eliminated  overruns  on 
defense  programs  during  the  recent  decade  had  it  been  available  and  applied.  It 
should  be  noted  that  when  Figure  9  is  in  fact  applied,  the  decision  maker  will  un¬ 
doubtedly  have  been  misinformed  as  to  what  fraction  of  the  program  is  actually 
complete.  This  distortion  has  already  been  compensated  for  in  Figure  9  using  Law 
Number  IV. 

A  word  of  caution  is,  of  course,  in  order  with  respect  to  the  delegation  of 
authority  for  the  management  of  the  contingency  funds  thus  determined,  lest 


FJGURE  9 

Predicting  Program  Cost  (R&D  Plus  Procurement] 
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Parkinson's  Law  exert  itself  and  costs  thereby  rise  to  meet  the  accessible  funds. 
Thus  we  have: 

Yll:  In  order  to  better  the  record  of  some  program  cost  estimators  of  the  past 
few  decades,  it  will  be  necessary  to  work  twice  as  hard;  to  be  twice  as 
smart;  and  to  recognize  unknown-unknowns.  Fortunately,  this  is  not  dif¬ 
ficult 

The  High  Cost  of  Buying 

Law  Number  VIII  addresses  the  prospect  that  warfare  is  pricing  itself  out  of 
existence. 

It  can  be  shown  that  the  unit  cost  of  military  equipment,  as  with  much  other 
high  technology  equipment,  is  increasing  at  an  exponential  rate.  Figure  10  shows, 
for  example,  the  historical  trend  of  rising  unit  cost  in  the  case  of  tactical  aircraft. 


FIGURE  10 

Trend  of  Increasing  Cost  of  Tactical  Aircraft 
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From  the  days  of  the  Wright  Brothers"  airplane  to  the  days  of  modem  high 
performance  fighter  aircraft,  the  cost  of  an  individual  aircraft  has  invariably 
grown  by  a  factor  of  four  every  10  years.  This  rate  of  growth  seems  to  be  an  in¬ 
herent  characteristic  of  such  systems,  with  the  unit  cost  being  most  closely  cor¬ 
related  with  the  passage  of  time  rather  than  with  changes  in  speed,  weight,  or 
other  technical  parameters.  The  same  inexorable  trend  can  be  shown  to  apply  to 
commercial  aircraft,  bombers,  helicopters,  or  even  ships  and  tanks,  although  in 
the  last  two  somewhat  less  technologically  sophisticated  instances,  the  rate  of 
growth  is  a  factor  of  two  every  10  years.  Seemingly  then,  the  cost  of  high 
technology  military  hardware  can  be  accurately  explained  in  terms  of  an  increase 
by  a  factor  of  four  during  each  sunspot  cycle,  independent  of  anything  else] 
The  significance  of  this  observation  does  not  however,  lie  in  the  mere  fact 
that  cost  growth  is,  in  itself,  predictable.  Rather,  it  lies  in  a  corfljMrrison  of  the 
rate  of  growth  of,  say,  aircraft  unit  cost  with  the  rate  of  growth  of  other  relevant 
parameters,  e,gM  the  defense  budget.  This  particular  comparison  is  presented  in 
Figure  11,  wherein  the  identical  data  points  shown  in  Figure  10  are  reproduced. 


FIGURE  11 


YEAH  OF  Bum  GPEfUTICliUL  tAMIUTY 

TWWlffAR  bOlLARt 


Defense  ARJ,  January  2015,  Vol.  22  No.  1:2-63 


21 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


64  |  Defeme  Systems  Management  Review 


but  to  a  smaller  scale  in  order  to  facilitate  extrapolation  into  the  future.  Objection 
might  be  raised  as  to  the  validity  of  any  such  extrapolation;  however,  it  is  noted 
that  the  above-mentioned  trend  has  faithfully  prevailed  throughout  the  history  of 
aviation r  presumably  making  such  extrapolation  no  more  hazardous  than  that 
used  in  most  other  fields  of  economic  forecasting. 

When  the  trend  curves  for  the  national  budget  for  defense  and  the  unit  cost  of 
tactical  aircraft  are,  in  fact,  extended  forward  in  time,  as  shown  in  Figure  II,  a 
rather  significant  event  can  be  predicted  for  the  not-t  co-distant  future.  Namely, 
the  curves  intersect.  And  they  intersect  within  the  lifetimes  of  people  Living  to¬ 
day.  This  observation  has  led  to  the  formulation  of  Augustine's  Eighth  Law; 

V11L  In  the  year  2054,  the  entire  defense  budget  will  purchase  just  one  tactical 
aircraft.  This  aircraft  will  have  to  be  shared  between  the  Air  Force  and 
Navy  3 Vi  days  each  per  week. 

One  can  only  imagine  the  difficulties  that  such  an  arrangement  will  entail. 
And  it  should  be  pointed  out  to  those  who  take  solace  in  challenging  the  validity 
of  the  above  extrapolation  of  the  defense  budget  that,  were  a  plot  of  the  gross  na¬ 
tional  product  to  have  been  used  instead,  the  above-mentioned  singular  event 
would  have  been  delayed  a  mere  60  years. 

This  particular  law  might,  perhaps,  more  accurately  be  remembered  as 
"Calvin  Coolidge's  Revenge"  as  a  tribute  to  the  prescience  of  that  gentleman.  It 
will,  of  course,  be  recalled  that  it  was  Calvin  Coolidge  who  once  asked,  in  a  mo¬ 
ment  of  budgetary  frustration  (which  now  can  be  quantitatively  understood 
"Why  can't  we  buy  just  one  aeroplane  and  let  the  aviators  take  turns  flying  it?" 

Turning  to  more  recent  events,  Figure  12  shows  the  trend  in  unit  cost  of 
bomber  aircraft,  culminating  in  the  B-l  phantom  data  point.  For  the  sake  of  con¬ 
sistency,  this  curve  can  be  referred  to  as  "Jimmy  Carter's  Revenge." 

On  Doing  Less  with  More 

Law  Number  IX  describes  how  one  can  make  a  silk  purse  out  of  a  sow's 
ear— if,  that  is,  one  starts  with  a  silk  sow. 

Although  some  types  of  systems  are  admittedly  expensive,  they  clearly  are 
also  much  more  effective.  Or  are  they?  One  such  comparison  can  be  made  by  ex¬ 
amining  the  combat  effectiveness  of  two  classes  of  military  systems  having  widely 
differing  costs— guided  missiles  and  guns,  Figure  13  plots  the  military  contribu¬ 
tion  of  these  two  categories  of  systems  during  various  major  conflicts  that  have 
taken  place  since  the  advent  of  the  missile  age.  In  each  of  the  conflicts  considered, 
both  types  of  systems  were  used  fairly  extensively,  thereby  providing  a 
reasonably  large  data  base.  The  combat  impact  of  each  category  of  system  is 
measured  in  terms  of  the  fraction  of  a  given  type  of  enemy  materiel  (airplanes, 
tanks,  etc.)  which  was  destroyed  by  missiles  or  guns,  respectively.  The  cost  of  the 
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missile  and  gun  systems  is  measured  in  terms  of  '  expendables"  only,  which  is,  of 
course,  an  oversimplification,  but  which  is  at  least  partially  justified  on  the  basis 
that  the  launchers  {aircraft,  ships,  or  the  gun  tube  itself)  are  reusable. 

It  might  be  presumed  that  the  data  points  in  such  a  comparison  would  ag¬ 
gregate  along  the  dotted  line  shown  in  Figure  12;  Le.,  the  more  that  one  pays  for  a 
system,  the  more  it  contributes.  Disappointingly,  the  actual  data  points  do  not 
behave  according  to  such  a  trend  at  alL  Instead,  they  cluster  into  two  distinct 
groups  as  far  from  the  expected  line  as  possible.  The  data  points  representing 
missiles  indicate  that,  at  least  to  date,  such  systems  have  had  relatively  less  im¬ 
pact  on  the  outcome  of  battles  than  have  the  far  less  costly  gun  systems.  This  is 
presumably  due  in  part  to  the  increased  susceptibility  to  countermeasures  of  the 
more  sophisticated  systems;  but,  more  importantly,  it  is  probably  due  to  the  fact 
that  as  equipment  grows  more  costly  it  can  be  afforded  in  far  lesser  quantities, 
thereby  sometimes  offsetting  the  hoped-for  improvement  in  individual-item  per¬ 
formance. 

The  next  law,  thereby  derived  with  a  good  deal  of  liberty  from  empirical 
evidence,  can  be  stated: 

DC:  It  as  true  that  complex  systems  may  be  expensive,  but  they  don't  contribute 
much. 
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FIGURE  13 

Impact  of  Various  Systems  in  Combat 


So  Simple  It  Can't  Be  Trusted 

Law  Number  X  concerns  the  testing  of  new  products  and  reflects  a  view  ex¬ 
pounded  by  one  Casey  Stengel,  late  of  the  New  York  Yankees  HTve  had  no  ex¬ 
perience  with  that  sort  of  thing,  and  all  of  it  has  been  bad." 

Were  one  to  examine  the  relationship  between  the  amount  of  testing  that  is  re¬ 
quired  of  a  newly  developed  item  and  the  complexity  of  that  item,  it  might  not  be 
unreasonable  to  expect  that  the  more  complex  the  item  the  more  testing  it  re¬ 
quires,  If,  for  example,  a  chart  were  made  showing  the  number  of  flight  tests  of 
various  missile  systems  against  some  measure  of  their  complexity,  the  trend 
thereby  observed  would  show  a  direct  correlation,  Le.,  a  line  sloping  upward  to 
the  right. 

Not  so.  Figure  14  presents  such  a  plot,  based  on  the  assumption  that  unit  cost 
is  a  reasonable  metric  of  "complexity/'  It  is  seen  that  the  correlation  is  inverse, 
sloping  ffownmard  to  the  right! 
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FIGURE  14 

Relationship  of  Missile  Complexity  [Cost) 
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The  amount  of  testing  required  thus  seems  to  be  more  nearly  explainable  in 
terms  of  tradition  than  in  terms  of  any  technical  rationale,  with  relatively  simple 
unguided  artillery  projectiles  somehow  demanding  thousands  of  test  rounds 
whereas  a  new  intercontinental  ballistic  missile  needs  only  a  few  test  flights  to 
demonstrate  its  adequacy.  The  less  complex  the  system,  the  more  testing  it  then 
requires,  a  consequence  of  which  forms  the  basis  of  Law  Number  X,  the 
August  ine^McKinley  Law;* 

Xs  Truly  simple  systems  are  not  feasible  because  they  would  require  near- 
infinite  testing. 

As  a  corollary  to  the  above  law,  it  will  be  noted  that  when  one  knows  the 
number  of  flight  tests  which  are  planned  in  a  missile  program,  one  may  use  Figure 
13  to  predict  the  uni#  cosf  of  the  item  in  question]  This  requires  only  a  few  man- 
seconds  of  labor  and  provides  results  that  compare  quite  favorably  in  terms  of  ac¬ 
curacy  with  the  official  cost  estimates  for  most  programs  during  the  past  two 
decades. 
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Going  NQwkerer  but  Making  Good  Time 

Law  Number  XI  follows  an  observation  made  by  a  well-known  college  foot¬ 
ball  coach:  "The  light  at  the  end  of  the  tunnel  may  be  a  freight  train," 

One  consequence  of  the  myriad  of  problems  examined  above  is  a  relatively 
high  mortality  rate  among  development  programs.  This  is  a  condition  which,  in¬ 
cidentally,  is  rarely  reflected  in  a  contractors  long-range  sales  plan,  but  which 
nonetheless  is  highly  predictable  in  the  aggregate. 

The  data  presented  in  Figure  15  are  derived  from  over  300  defense-related  pro¬ 
grams  conducted  in  the  past  two  decades.  They  reveal  the  probability  that  any 
given  program  will  fail  to  survive  the  threats  to  its  existence  which  arise  prior  to 
any  given  year  in  its  life.  It  is  seen  that  there  is  about  a  4-percent  probability  of 
cancellation  of  a  program  each  and  every  year  except  for  the  first  year,  sometimes 
referred  to  as  the  honeymoon  period.  This  probability  appears  to  be  relatively  in¬ 
dependent  of  program  age,  presumably  even  for  such  aged  endeavors  as  two  cur¬ 
rent  programs  which  soon  will  have  been  in  development  for  IS  years. 

XI:  In  terms  of  their  chances  of  surviving,  most  programs  start  out  kmd  of 
slowly  and  then  sort  of  taper  off. 


FIGURE  15 

Survival  Expectancy  of  Development  Programs 
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Do  They  Know  Something  We  Don  t? 

Law  Number  XII  examines  a  process  similar  to  one  once  critically  referred  to 
by  an  executive  of  the  Lockheed  Aircraft  Corporation  as  "You  bet  your  com¬ 
pany/' 

One  of  the  most  effective  means  of  controlling  cost  while  achieving  good  pro¬ 
duct  performance  is  competition.  But  even  competition  must  be  applied  carefully, 
or  unwanted  results  occur.  Consider,  for  example,  the  practice  occasionally  used 
in  Department  of  Defense  procurements  of  awarding  the  production  contract  for 
a  newly  developed  system  to  whomever  is  the  low  bidder.  This  has  the  unques¬ 
tioned  impact  of  driving  down  the  bid  prices,  and  the  disadvantage  of  sometimes 


FIGURE  16 

Effect  of  Competition  on  Unit  Price 
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creating  a  producer  who  has  no  familiarity  with  the  hard-earned  lessons  of  how 
one  actually  goes  about  producing  the  product  in  question,  lessons  which  were 
learned  over  many  years  of  effort  by  the  developer. 

The  data  in  Figure  16  verify  that  major  bid  price  reductions  are  indeed  ob¬ 
tained  by  competing  a  number  of  potential  producers  for  an  item  developed  by 
one  specific  contractor,  (This  figure  does  not,  however,  examine  whether  the 
winning  bidder  was  ever  able  to  actually  manufacture  a  useful  and  reliable  end 
product  at  the  bid  price—or  any  other  price,) 

Figure  17  examines  the  data  in  Figure  16  in  a  slightly  different  fashion.  It  in¬ 
dicates  that  the  greater  the  winning  price  reduction  relative  to  the  developer's 
original  price,  the  less  likely  is  the  developer  of  the  item  in  question  to  be  the  win¬ 
ning  bidder.  It  appears  that  an  intimate  knowledge  of  the  task  to  be  performed  is 
somehow  a  handicap.  Several  interpretations  are,  of  course,  possible,  one  of 
which  is  expressed  thusly  (with  apologies  to  Alexander  Pope); 

XU;  Fools  rush  Ln  where  incumbents  fear  to  bid. 


figure  i7 

The  Ability  of  an  Established  Producer  to  Win  a  Breakout  Competition 
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It  was  this  law  which  an  Army  aviator,  with  whom  the  author  once  flew,  had 
in  mind  when  he  added  to  the  warning  and  caution  stickers  that  traditionally 
abound  in  the  cockpits  of  modern  rotary-wing  aircraft,  this  hand-lettered  ad¬ 
monition'  "Caution,  This  helicopter  built  by  the  lowest  bidder /' 

The  Budget  Equation 

Law  Number  XIII  concerns  the  impact  of  the  congressional  appropriations 
process  in  defense  system  management. 

In  order  to  survive  to  completion,  every  government  development  program 
must  maintain  an  extremely  high  single-skirmish-survival  probability  in  its  en¬ 
counter*  with  the  various  steps  in  the  budget  cycle.  In  the  congressional  approval 
process  alone,  a  defense  program  s  budget  will  be  voted  on  at  least  18  times  a 
year,  or  a  total  of  144  times  in  the  average  program's  lifetime.  It  does  not  seem  to 
be  possible  to  determine  a  priori  the  probability  that  any  particular  program  will 
be  funded  or  terminated  by  the  Congress  in  any  given  year.  It  is,  however,  possi¬ 
ble  to  predict  with  very  good  accuracy  what  the  overall  impact  of  the  congres¬ 
sional  approval  process  will  be  on  the  defense  budget;  that  is,  the  result,  in  the  ag¬ 
gregate,  of  the  yearly  congressional  review  process  can  be  reduced  to  an  equa¬ 
tion. 

Figure  18  displays  the  effect  of  congressional  actions  on  the  administration's 
defense  budget  requests  in  each  year  of  the  present  decade.  It  is  seen  that  a  trend 
line  can  be  quite  accurately  drawn  which  will  predict  the  outcome  of  the  congres¬ 
sional  review  process  on  the  budget  of  any  given  military  department,  or  on  the 
Department  of  Defense  as  a  whole.  This  would  suggest  that  the  Administration's 
efforts  to  gain  approval  of  its  budget  requests  have  about  the  same  impact  year 
after  year,  independent  of  the  political  parties  involved  or  the  magnitude  of  the 
budget  change  requested,  the  latter  even  over  quite  large  excursions. 

These  observations  are  summarized  as  follows; 

XII I;  In  any  given  year,  the  Congress  will  appropriate  for  defense  the  amount  of 
funding  approved  the  prior  year  plus  three-fourths  of  whatever  change  the 
administration  requests,  minus  a  4-percent  fcax+ 

During  the  present  decade,  this  law  has  applied  with  good  accuracy  over  a 
ange  of  year-to-year  changes  in  the  requested  funding  level  extending  from 
minus  7  percent  to  plus  24  percent.  This  is  shown  in  the  above-mentioned  figure. 

On  Making  a  Precise  Guess 

Law  Number  XIV  examines  the  parallel  between  management  decision- 
making  and  Bismarck's  observation  about  law-making.  'Law/'  he  said,  "is  like 
sausage;  if  you  like  it,  you  shouldn't  watch  it  being  made." 

As  reported  to  the  Congress  at  the  time  development  was  to  be  initiated,  the 
total  program  cost  for  the  Harpoon  missile  program  was  said  to  be  $1,031.8 
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FIGURE  16 

Predicting  Congressional  Changes  to  the  Defense  Budget  Request 


A 


million.  For  the  A-10  program r  the  corresponding  cost  was  defined  as  $2,48^7 
million— not  $2 r 400  million;  not  even  $2,489  million.  Rather,  the  cost  would  be 
two  thousand  four  hundred  eighty-nine  paint  seven  million  dollars. 

This  great  degree  of  accuracy  may  perhaps  be  somewhat  surprising  to  the 
uninitiated  in  view  of  the  fact  that  history  shows  the  first  digit  of  past  program 
cost  estimates  to  have  been  in  error,  on  the  average,  by  about  100  percent!  The 
General  Accounting  Office,  in  its  most  recent  report  on  the  topic,  for  example, 
has  stated  that  for  Department  of  Defense  acquisition  programs  now  underway, 
67  percent  are  already  overrun  by  more  than  100  percent, 

Nonetheless,  by  examining  the  data  in  Figure  19,  it  is  possible  to  derive  the 
logic  which  underlies  the  practice  of  quoting  fundamentally  dubious  numbers 
with  a  very  great  degree  of  apparent  accuracy.  It  is  seen  from  the  figure  that  there 
is  indeed  a  relationship  between  the  number  of  "significant  figures"  quoted  and 


30 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


Augustine' &  Laws  If  73 


FIGURE  19 

Relationship  of  Implied  Precision  to  Actual  Precision 
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the  true  precision  of  the  data  at  hand  but  this  relationship  is  just  the  opposite  of 
what  one  might  expect.  My  next  law,  which  is  based  on  a  substantive  collection 
ol  data  such  as  that  presented  in  Figure  19,  states: 

XIV:  The  weaker  the  data  available  upon  which  to  base  one's  position,  the 
greater  the  accuracy  which  should  be  quoted  in  order  to  give  that  data  an 
aura  of  authenticity. 

A  problem  which  has  long  been  faced  in  applying  Law  Number  XIV, 
however,  has  been  what  to  do  in  those  cases  wherein  the  analyses  from  which  the 
numbers  were  derived  provide  only  rather  discrete  results,  such  as  $1  billion,  or 
10  miles  or  1  ton.  The  solution  to  this  dilemma  has  been  astutely  observed  by 
Lieutenant  General  Glenn  A,  Kent  (USAF,  Ret.)  in  his  reviews  of  a  large  number 
of  quantitative  analyses.  The  solution  consists  simply  of  converting  all  data  from 
the  English  system  of  measures  into  the  metric  system  and  back  again f 

A  derivative  of  this  technique  accounts  for  such  phenomenal  accuracies  as  are 
identified  in  a  bulletin  recently  carried  by  a  U.S.  wire  service  concerning  a  British 
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citizen  whose  private  airplane  was  reported  to  have  missed  crashing  into  the  con¬ 
trol  tower  at  an  airport  in  England  "by  less  than  39,4  inches." 

A  related  approach  appears  to  have  been  used  last  year  in  testimony  provided 
to  the  Congress  by  the  General  Accounting  Office  in  which  it  was  stated  that  the 
chances  of  a  person  on  the  ground  being  injured  by  a  falling  piece  of  a  space  shut¬ 
tle  launched  in  a  northerly  direction  from  Kennedy  Space  Center  "are  one  in 
166,667/'  It  may  or  may  not  be  coincidence  that  one  change  in  166,667  equates 
almost  precisely  to  6  divided  into  a  million.  But  clearly,  one  would  not  feel  nearly 
as  safe  knowing  that  the  chances  of  being  hit  on  the  head  by  a  falling  piece  of 
shuttle  are  about  “half  a  dozen  in  a  million"  as  he  feels  when  the  probability  of 
that  happening  is  a  single  chance  in  166,667. 

Still  another  approach  underlies  the  fiscal  year  1979  appropriation  of  $25,418 
million  for  the  Aegis  program.  Certainly,  a  great  deal  of  detailed  study  must  have 
been  required  to  define  the  program's  funding  needs  in  such  detail.  But,  alas, 
when  scrutinized  more  closely  it  is  found  that  the  figure  is  the  result  of  a  com¬ 
promise  brought  on  by  a  dispute  between  the  House  and  Senate  whereby  a  lump 
sum  of  $11  million  was  simply  patched  on  top  of  the  original  request  by  the  Presi¬ 
dent,  which  was  for  $14,416  million  I 

Actually,  Sir  fogiah  Stamp,  Her  Majesty's  Collector  of  Inland  Revenue,  was 
well  on  the  track  of  Law  Number  XIV  nearly  a  century  ago,  except  that  he  applied 
it  only  to  government  and  neglected  Its  frequent  use  by  Industry,  among  others. 
Sir  Stamp  pointed  out  that;  rThe  Government  are  (sic]  extremely  fond  of  amass¬ 
ing  great  quantities  of  statistics.  These  are  raised  to  the  rjth  degree,  the  cube  roots 
are  extracted,  and  the  results  are  arranged  into  elaborate  and  impressive  displays. 
What  must  be  kept  ever  in  mind,  however,  is  that  in  every  case,  the  figures  are 
first  put  down  by  a  village  watchman,  and  he  puts  down  anything  he  damn  well 
pleases/' 

Growing  Like  a  Regulation 

Law  XV  provides  the  mathematical  foundation  for  Lamermais's  apothegm, 
which  states:  “Centralization  breeds  apoplexy  at  the  center  and  anemia  at  the  ex¬ 
tremities/' 

Large  organizations,  probably  epitomized  by  federal  governments,  seem  to  be 
susceptible  to  the  concept  that  regulations  can  become  a  substitute  for  manage¬ 
ment.  Today,  for  example,  the  U.S.  Government  has  imposed  a  set  of  23,000 
specifications  on  those  who  would  sell  to  it  a  simple  mousetrap.  But,  in  spite  of  all 
the  established  rules,  it  is  soon  discovered  that  special  cases  occur,  each  requiring 
still  further  rules.  And,  of  course,  as  new  rules  are  added,  none  of  the  old  rules  is 
ever  discarded;  none,  that  is,  until  the  entire  management-by-regulation  concept 
collapses  of  its  own  weight  and  a  new  cycle  begins  based  on  an  altogether  new  set 
of  regulations. 
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One  particularly  interesting  but  not  atypical  example  of  the  growth  of  regula¬ 
tions  is  to  be  found  in  the  Armed  Services  Procurement  Regulation  (ASPR)  which 
governs  procurement  of  everything  used  in  the  nation's  defense,  from  aircraft  car¬ 
riers  to  the  paper  on  which  the  ASPR  itself  is  printed.  Figure  20  shows  the  rate  of 
growth  of  ASPR  over  its  lifetime,  and  verifies  that  it  indeed  exhibits  a  behavior 
consistent  with  well  established  growth  processes  observed  in  nature.  It  is  also 
noted  that,  based  solely  on  its  growth  pattern,  the  ASPR  appears  to  have  reached 
its  terminal  phase —after  which  it  can  be  expected  to  be  replaced  by  a  new  set  of 
policies. 

The  degree  of  Improvement  wrought  by  these  growth  trends,  as  they  have 
prevailed  over  the  years,  is  suggested  in  the  case  histories  of  two  airplanes.  When 
the  Army  Signal  Corps  purchased  the  development  of  an  aircraft  from  the  Wright 
Brothers,  the  entire  contract  (a  fixed-price  incentive  type)  comprised  two  pages,  It 
was  the  result  of  a  40-day  competition  among  41  bidders,  which  was  culminated 
in  a  9-day  evaluation  period  by  the  government.  An  award  was  made  (without 


FIGURE  20 
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protest)  and  the  aircraft  flew  successfully  &Yi  months  later.  The  primitiveness  of 
this  management  system  contrasts  sharply  with  the  more  sophisticated  approach 
used  today  which,  in  the  case  of  the  giant  C-5A  transport,  generated  contractor 
proposals,  the  paper  for  which  would  have  more  than  filled  the  C-5A  itself.  We 
are  thus  led  to: 

XV:  Systems  of  regulations  created  as  a  management  surrogate  take  on  a  life  of 
their  own  and  exhibit  a  growth  history  which  closely  parallels  that  of 
selected  other  living  entities  observed  in  nature. 

Summary 

Augustine's  Laws,  largely  derived  from  empirical  evidence,  might  be  inter¬ 
preted  as  suggesting  that  it  is  simply  not  possible  to  develop  major  systems.  Such 
is  not  the  case  This  is  demonstrated  by  the  many  successes  achieved  by  many 
able  and  dedicated  managers.  What  they  do  suggest  is  that  the  wntrary  manager 
will  very  likely  fall  victim  to  phenomena  which  are  every  bit  as  predictable  r  and 
every  bit  as  invincible,  as  processes  governed  by  the  physical  laws  of  nature.  | 
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Augustine’s  Laws 
and  Major  System 
Development  programs 

(Confirmed) 

Norman  R .  Augustine 

(The  Defense  Systems  Management  College  published  the  first  installment  of 
'AugH5fin£rj  Laws  and  Major  System  Development  Programs' "  (Laws  l-XV)  in 
the  Spring  1979  Issue  of  the  Defense  Systems  Management  Review  (VoL  2r  Nor 
2)+  Since  then  aufftor  fists  added  eight  more  "laws,  "  which  we  are  pleased  to 
make  available  to  Concepts  readers.  The  American  Institute  of  Aeronautics  and 
Astronautics  plans  to  publish r  in  fhe  near  future r  compete  collection  of  23 
laws  in  hardbound  form.  This  installment  of  rrAirgti5fiti£S  Laws' '  takes  up  where 
the  first  left  off—  at  Law  XVI  and  Figure  22. 
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The  Manager  of  the  Year 

When  the  Rising  gets  tough „  everyone  leaves. 

—Lynch's  Law 

Law  Number  XVI  addresses  the  problem  of  management  turnover  and  is 
premised  on  the  possibility  that  most  managers,  when  dealing  in  a  variety  of 
endeavors,  think  they  know  their  capacity  but  simply  pass  out  before  reaching  it. 

Certainly,,  one  of  the  greatest  impediments  to  that  fundamental  precept  of 
management  referred  to  as  "accountability”  is  the  rapid  turnover  of  individuals 
holding  leadership  positions.  Government  program  managers  in  the  acquisition 
process,  for  example,  hold  their  jobs  an  average  of  only  30  months.  Even  this  is  a 
substantial  improvement  over  the  situation  which  existed  just  a  few  years  ago,  in 
1965,  when  such  managers  retained  their  jobs  an  average  of  only  15  months. 
Over  the  last  two  decades  the  tenure  of  the  secretaries  of  the  military  departments 


©  19S1  by  K  R  AmouKiAt 


Authors  notet  "Augustine's  Laws/"  although  sometimes  writ  ten  in  a  humorous  vein  to  enhance 
readability  and  return  iveness,  deal  with  real  and  serious  matters.  3  offer  them  in  a  positive  and  con¬ 
structive  sense  with  the  hope  that  they  will,  by  drawing  attention  to  problems  and  some  of  I  heir 
potential  solutions,  assist  in  al  leas!  some  small  way  toward  the  strengthening  of  our  nation's  defense 
capability. 
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and  the  Secretary  of  Defense  as  a  group  has  been  no  better,  also  averaging  about 
30  months.  The  consequences  of  this  anonymity  in  responsibility  once  prompted 
an  aggrieved  Lyndon  Johnson  to  remark,  in  response  to  a  question  by  a  reporter 
as  to  why  he  had  not  fired  the  individual  who  had  scuttled  one  of  the  President's 
favorite  programs:  "Fire  him?  Hell,  I  can't  even  find  him." 

Could  it  be  possible  that  so  important  a  management  tenent  as  leadership 
stability  and  accountability  has  been  totally  overlooked  in  managing  our  nation's 
defense  affairs?  No,  there  is  reason  for  optimism.  Consider  the  following 
newspaper  article  quoting  senior  Navy  managers:  '“By  constantly  changing 
our  .  .  ,  director  every  two  or  three  years,  we  have  destroyed  continuity.  If  you 
had  a  million-and-a-half-dollar  business,  would  you  want  to  change  bosses  every 
three  years  for  someone  who  didn't  have  any  experience?  Most  directors  come 
right  from  sea  duty  to  this  job,  and  it  can  take  a  full  year  to  get  to  know  the  ropes. 
How  many  people  in  the  Navy  do  you  think  know  things  like  scheduling  prob¬ 
lems?" 

Encouraging  indeed;  The  problem  is  recognized.  But  is  this  an  article  from  the 
pages  of  The  Wall  Street  Journal  discussing  the  management  of  an  important  new 
Navy  fighter  aircraft,  or  perhaps  even  a  new  shipbuilding  program?  Alps,  the  ar- 
tide  is  from  the  sports  pages  of  The  Washington  Post,  addressing  the  decision 
reached  a  few  years  ago  to  stop  rotating  individuals  through  the  position  of 
athletic  director  at  the  Naval  Academy.  At  least  we  have  our  priorities  in  perspec- 
tive. 

Gitbert  Fiuhugh,  Chairman  of  the  Blue  Ribbon  Defense  Panel  of  the  late 
I960sr  stated  the  situation  in  the  following  terms:  "Everybody  is  somewhat 
responsible  for  everything,  and  nobody  is  completely  responsible  for  anything." 
A  two-star  general  once  commented,  in  an  outburst  of  candor  in  response  to  a 
question  as  to  how  he  was  going  to  work  his  program  out  of  a  seemingly 
untenable  position  into  which  it  had  descended:  "Perhaps  a  miracle  will  happen, 
or  etse  maybe  111  get  transferredf11 

This  problem  of  personnel  turbulence,  troublesome  in  virtually  all  manage¬ 
ment  situations,  is  particularly  acute  in  the  case  of  major  research  and  develop¬ 
ment  undertakings.  Consider  the  fact  that  studies  of  the  frequency  of  reference  to 
technical  articles  held  in  libraries,  and  of  the  change  of  content  of  course  catalogs 
m  the  scientific  departments  of  universities,  indicate  that  the  half-life  of  many 
technologies  is  today  only  about  10  yeans. 

Paraphrasing  this  inconsistency  as  once  pointed  out  by  the  Armed  Forces 
Journal  International,  we  are  attempting  to  develop  major  new  systems  with 
10-year  technology ,  8-year  programs,  a  5-year  plan,  3-year  people,  and  1-year 
dollars. 
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The  evidence  which  underlies  Law  Number  XVI  is  presented  in  Figure  21 , 
wherein  the  longevity  of  program  managers  is  compared  with  the  (average) 
longevity  of  the  programs  they  manage.  As  also  shown  in  the  figure,  the  people  at 
the  top  of  the  legislative  structure  experience  relatively  little  turnover.  These 
numbers  of  the  legislative  branch  not  infrequently  remind  Defense  witnesses  testi¬ 
fying  before  R&D  hearings  that  the  congressmen  and  senators  themselves  know 
more  about  the  history  and  underlying  problems  of  the  programs  in  question 
than  does  the  parade  of  so-called  experts  who  appear  before  them  year  after  year 
with  ever-greater  enthusiasm  and  optimism.  It  is  just  this  dichotomy,  aggravated 
by  the  very  length  of  the  acquisition  process,  which  in  fact  leads  to  the  Law  of 
Limited  Liability^ 

XVI:  The  problem  with  the  acquisition  process  is  that  by  the  time  the  peo¬ 
ple  at  the  top  are  ready  for  the  answer,  the  people  at  the  bottom 
have  forgotten  the  question. 


FIGURE  21 

Personnel  Stability  in  Acquisition  Process 


TOTAL  EXPERIENCE  AT  TIME  OF  SEPARATION 
F I V  E-POINT  FI  DL  L I  NO  A  V  Efl  AO  ES 


rM 


3 

O 

<E 

111 

4k 

z 

o 

p 

(tt 

X  Q 


sc 

S3 

»  Ui 

K  > 
IV  uj 

a 


1.0" 


MEMBERS  OF  US.  SENATE 


MEDIAN  DURATION  OF  DEVELOPMENT  PROGRAMS  OVER  PERIOD  SHOWN 


"10 


S  M’S 

K 

D 

I 

laJ 

S 


GOVERNMENT  PROGRAM  MANAGERS 


1965 


1W 


19*S 


VI AR  Of  TERMINATION 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


Augustine's  Lam 


Malice  in  tVonefer/jaMd 

Bit!  Benjamin's  mess  was  Fi&e  limes  so  much  as  any  of  theirs. 

—Book  of  Genesis 

Law  Number  XVII  examines  the  incentive  system— and  demonstrates  that 
managers  who  produce  exceptional  results  can  expect  the  rewards  they  receive  to 
be  increased.  Unless,  of  course,  those  rewards  stay  the  same  or  go  down, 

'Call  it  what  you  will,  Incentives  are  the  only  way  to  make  people  work 
harder/1  The  words  of  Andrew  Carnegie?  The  creed  of  John  D.  Rockefeller?  Or 
perhaps  of  Henry  Ford?  No,  as  it  happens,  these  are  the  words  of  none  other  than 
that  old  capitalist  Nikita  Khruschev  speaking  on  the  benefits  of  the  incentive  pro¬ 
gram* 

Having  thus  established  the  manner  in  which  incentives  are  viewed  in  the 
Soviet  Union,  it  is  instructive  to  examine  their  use  in  the  economic  system  extant 
in  the  United  States,  for  which  incentives  form  the  very  foundation— the  "free- 
enterprise  system." 

Figure  22  displays  the  ranking  of  the  50  most  profitable  firms  in  the  United 


FIGURE  22 

Relationship  of  Executive  Wages  to  Company  Performance 
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States  in  1978  as  compared  with  the  rank,  according  to  pay  received  of  the  indi¬ 
viduals  who  led  those  companies  the  prior  4  years.1  The  following  law,  known  as 
the  Augustine-Lemeshewsky2  Law  of  Distributive  Rewards,  explains  the  evidence 
in  Figure  22  (with  apologies  to  P.  K.  Wrigley  of  baseball  fame): 

XVEI:  There  are  many  highly  successful  organizations  in  the  United  States, 
There  are  also  many  highly  paid  executives.  The  policy  is  not  to  in¬ 
termingle  the  two. 

If  a  plot  is  made  showing  rankings  according  to  retum-on-equity,  the  lack  of 
correlation  exhibited  is  even  more  striking  than  that  found  in  Figure  22  for  abso¬ 
lute  profits.  The  evidence  seems  to  be  incontrovertible. 

Further,  although  one  could  never  confuse  the  operation  of  the  U,S,  govern¬ 
ment  with  the  free-enterprise  system,  it  is  still  striking  that  an  oyer*  effort  at 
demotivation  ha?  been  practiced  whereby  the  top  live  layers  of  management  have 
all  been  fixed  at  the  identical  pay  level  due  to  the  imposition  of  an  apparently  ar¬ 
bitrary  wage  celling, 

T/i£  Half-Life  of  a  Manager 

We  have  a  lot  of  players  in  their  first  year.  Some  of  them  are  in  their  last 
year, 

—Bill  Walsh,  Coach,  San  Francisco  49ers 

Law  Number  XVI II  examines  the  viewpoint  expressed  by  former  Dallas 
Cowboy  Guard  Blaine  Nye:  "It's  not  whether  you  win  or  lose  that  counts/'  he 
says,  "but  who  gets  the  blame/'  Will  Rogers  once  pointed  ouL  with  respect  to  his 
business  pursuits,  "It  is  not  the  return  on  my  investment  that  I  am  concerned 
about;  it  is  the  return  of  my  investment/'  Perhaps  within  this  philosophy  lies  the 
key  to  refute  the  rather  disappointing  thrust  of  the  previous  law. 

Possibly  the  significant  consideration  with  respect  to  successful  managers  is 
not  what  they  get  from  their  jobs,  but  rather  that  they  get  to  keep  their  jobs.  This 
possibility  can  be  readily  assessed  using  Figure  23,  which  displays  the  number  of 
years  the  top  executive  in  the  20  most  profitable  firms  in  the  United  States,  in  re¬ 
cent  years,  has  been  able  to  hold  his  job — as  a  function  of  the  success  achieved  by 
that  executive  in  increasing  the  company's  profits,  Unfortunately,  the  results  are 
doubly  disappointing.  Not  only  do  they  fail  to  refute  Law  Number  XVI I r  but 
worse  yetr  they  call  for  still  another  law,  the  Law  of  Infinite  Mortality: 


1.  For  I  he  occasion  inst  antes  where  I  he  leadership  changed  during  the  period  examined,  the  data 
for  that  company  are  not  inducted  in,  ihe  figure. 

2.  Susan  D.  Lemeshewsky:  Technical  Operations  Intern,  Martin  Marietta  Aerospace. 
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Figure  ?a 

Executive  Survivability 
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XVIII:  Executives  who  do  not  produce  successful  results  can  be  expected  to 
hold  on  to  their  jobs  only  about  five  years.  On  the  other  hand,  those 
who  do  produce  effective  results  can  expect  to  hang  on  about  half  a 
decade. 

It  should  be  possible  to  fight  this  form  of  apathy;  but  so  far  It  has  not  been 
possible  to  find  anyone  interested  enough  to  do  so.  The  conclusion  of  the  above 
law  seems  to  be  true  over  a  wide  span  of  profit  growth  and  even  over  severe  prof¬ 
it  ■"retrenchments/"  as  they  are  gently  referred  to  in  stockholders'  reports.  The 
correlation  coefficient  between  profit  growth  and  the  ability  to  retain  one's  job, 
on  a  scale  where  0  is  purely  random  and  1,0  represents  perfect  correlation,  is 
calculated  to  be  0,1,  The  strongest  correlation  observed  between  longevity  and 
any  other  parameter  examined  is  found  to  be  between  the  first  letter  in  the  name 
of  the  company  and  the  first  letter  in  the  last  name  of  the  chief  executive;  as  in 
"Ford,  Henry  il 

A  median  survival  duration  of  a  little  over  5  years  for  top  executives  may 
seem  rather  short  at  first  glance/  however,  it  is  really  quite  good  when  compared 

3,  The  data  sample-  considered  in  Figure  23  contains  a  slight  potential  bias  since  the  available 
evidence  cover*  only  a  1  fry  ear  period.  The  impact  of  this  is  lo  have  relatively  little  effect  on  (he  me¬ 
dian  longevity  addressed  herein;  however,  the  overall  (arithmetic}  average  longevity  would  perhaps 
increase  to  7  or  fl  years. 
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with  many  other  professions  such  asr  say,  coaching  football.  Consider  the  case  of 
the  Washington  Redskins  coach,  who,  several  years  ago,  was  fired  at  half-time  of 
the  first  exhibition  game?  or  the  situation  that  developed  a  few  years  later  when 
the  team  had  three  head  coaches  in  24  hours.  In  fact,  in  pro  football  it  is  dearly  a 
liability  to  be  recognized  for  outstanding  performance.  Of  the  last  15  coaches  to 
be  honored  by  the  Associated  Press  as  coach  of  the  year,  11  were  fired  within  the 
next  12  months.  As  Bum  Phillips,  coach  of  the  Houston  Oilers,  notes,  'There's 
only  two  kinds  of  coaches,  them  that's  been  fired  and  them  thatJs  about  to  be 
fired. "  (Phillips  has  since  been  fired  by  Houston). 

It  would  seem  that  to  err  may  in  fact  be  human,  but  to  forgive  is,  by  and 
large,  against  company  policy.  As  John  McKay,  coach  of  the  Tampa  Bay  Bucca¬ 
neers  points  out,  'They're  paid  to  catch  the  ball/'  It  can,  of  course,  be  asserted 
that  many  of  the  individuals  included  in  the  data  base  of  Figure  23  retired  or 
moved  on  to  more  important  jobs.  But  it  can  be  equally  accurately  asserted  that 
many  of  these  individuals  were  yet  relatively  young  at  the  time  of  their  depar¬ 
tures  and  already  enjoyed  some  of  the  best  jobs  in  America. 

Nonetheless,  there  is  no  need  for  discouragement,  since  the  incentive  system 
is,  in  spite  of  the  above  evidence,  still  alive  and  well:  "People  who  show  the  best 
example  in  their  work  must  receive  greater  material  benefit" — according  to  a 
speech— before  the  Supreme  Soviet— by  the  Premier— of  the  U.$-$,R,  And  right 
here  at  home  it  was  recently  pointed  out  that  "the  challenge  for  American 
capitalism  in  the  is  to  bring  the  entrepreneurial  spirit  back  to  America. 
Depressed  areas  especially  need  an  enormous  investment  of  capital.  Individual 
entrepreneurship  can  create  the  new  work  ethic  that  is  so  desperately  needed  in 
America.  To  stimulate  that  ethic  America  needs  creative  financing  .  .  .  and  I  in¬ 
tend  to  work  to  create  it;" 

So  said  Jerry  Rubin,  Yippie  leader  of  the  1960s  and  a  defendant  in  the  Chicago 
Seven  trial— speaking  in  the  1970s  as  a  security  analyst  on  Wall  Street, 

The  Reliability  of  Unreliability 

Adde  Parvum  Parvo  Magnus  Acervus  Eiit.* 

—Ovid 

The  following  law  deals  with  the  relationship  between  the  .reliability  of  com¬ 
plex  hardware  and  that  human  tendency  reflected  in  the  World  War  II  placebo: 
"We  know  of  not  a  single  instance  wherein  the  enemy  has  successfully  used 
camouflage  against  us;"  However,  with  respect  to  the  matter  of  enhancing 


4.  Add  little  to  little  and  there  witl  be  a  big  pile."1  Quoted  frtutt  The  Mythical  Man  Month  by 
Frederick  P.  Brooks  Jr. 
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reliability,  we  have  in  fact  u  jewed  the  enemy;  and,  to  once  again  quote  that  im¬ 
mortal  possum,  "he  is  us/' — but  we  seem  unwilling  to  recognize  this  is  the  case, 

It  does  appear  to  be  fundamental  to  the  human  race  to  believe  that  which  one 
wants  io  believe  rather  than  that  which  a  logical  examination  of  fact  would 
reveal.  George  Santayana  put  it  in  the  following  terms:  "All  living  souls  welcome 
what  they  are  ready  to  cope  with;  all  else  they  ignore  or  pronounce  to  be 
monstrous  and  wrong  ...  or  deny  to  be  possible."  It  would  appear  that  Mark 
Twain  may  have  been  unduly  generous  toward  humanity  when  he  speculated,  "I 
believe  that  our  Heavenly  Father  invented  man  because  He  was  disappointed  in 
the  monkey," 

Consider  the  crucial  matter  of  producing  more  reliable  hardware;  and,  in  par¬ 
ticular,  electronic  hardware,  which  everyone  knows  comprises  components 
whose  individual  reliabilities  have  been  improving  at  a  rate  of  about  15  to  20  per¬ 
cent  per  year  for  nearly  two  decades.  Further,  with  size  decreasing  dramatically 
and  the  aggregate  cost  of  integrated  circuits  consistently  decreasing  since  1963 
along  a  75  percent  learning  curve,  it  should  be  possible  to  achieve  extraordinary 
system  reliability  through  careful  component  selection  and  built-in  redun¬ 
dancy— and  thus  to  eliminate  what  has  been  one  of  the  most  troublesome  prob¬ 
lems  for  electronic  equipment  users  for  many  years:  unreliability. 

In  the  words  of  Lieutenant  General  Orwin  Talbott,  rThe  longer  a  man  is  in  a 
command  position  on  the  battlefield,  the  less  enamored  he  is  of  the  technological 
edge  and  the  more  obsessed  he  becomes  with  trying  to  make  what  he  has  work." 

Now,  if  one  were  not  privy  to  the  anachronistic  behavior  of  engineering  and 
management  activities  as  they  have  been  dissected  herein,  one  might  in  fact  un¬ 
wittingly  conclude  that  as  more  and  more  money  is  spent  on  an  item,  its  reliabil¬ 
ity  would  get  progressively  better  and  better,  The  initiated  would  never  fall  into 
such  a  logical  trap  and  would  recognize  immediately  that  quite  the  opposite  must 
be  true.  That  this  latter  situation  indeed  prevails  is  verified  in  Figure  24,  which  ex¬ 
hibits  field  reliability  data  on  a  number  of  airborne  electronic  systems  as  collected 
during  the  Hlectronics-X  study  conducted  under  the  aegis  of  the  Institute  for 
Defense  Analyses,  It  is  seen  that  the  items  examined  range  from  relatively  simple 
devices  such  as  marker  beacons  and  glide-slope  indicators,  to  completely 
automated  multichannel  airborne  intercept  systems.  The  costs  and  reliability  fac¬ 
tors  change  with  increases  in  inflation  and  technology — but  the  trend  at  any 
given  time  remains  unwavering.  Whatever  the  spectrum  of  equipment  and  tech¬ 
niques  involved,  the  conclusion  is  unmistakable:  As  cost  increases,  reliability 
does  not  improve;  rather,  it  worsens.  Frank  McKinney  Hubbard  {186B-193Q  ad¬ 
vises,  rrff  at  first  you  do  succeed r  quit  trying,"  This  is  summarized  in  the  Law  of 
Undiminished  Expectations: 
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FIGURE  24 

Impact  of  Increasing  Unit  Cost  on  Field  Reliability 


AIRBORNE  ELECTRONICS  EQUIPMENT 


XIX:  It  is  very  expensive  to  achieve  high  degrees  of  unreliability.  It  is  not 
uncommon  to  increase  the  cost  of  an  item  by  a  factor  of  ten  for  each 
factor  of  ten  degradation  accomplished. 

Dr.  Eb  Rechtin,  President  of  The  Aerospace  Corporation,  points  out  that  such 
has  been  the  pace  of  technological  progress  that  by  spending  $250  million  for  an 
item,  a  mean  time  between  failure  of  30  seconds  can  be  guaranteed.  Correspond¬ 
ingly,  one  might  suspect  that  a  mean  time  to  repair  of  30  months  could  be  suf¬ 
fered. 

Although  great  care  must,  of  course,  be  taken  in  interpreting  the  meaning  of  a 
"failure'"  {ail  failures  are  not  created  equal,  nor  do  they  have  equal  consequences!, 
data  released  on  the  mean  flight  hours  between  failure  for  12  different  types  of 
Navy  and  Air  Force  fighter  and  attack  aircraft  are  illuminating.  Nine  of  the  12 
aircraft  experienced  a  “failure"  at  least  once  every  30  minutes.  Of  those,  five  ex- 
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pericncd  failures  every  15-20  minutes.  This  would  seem  to  be  conclusive  proof 
of  the  correctness  of  those  who  have  argued  that  the  next  strategic  bomber  must 
be  supersonic  rather  than  subsonic. 

In  any  event  it  can  be  understood  why  there  are  those  who  say  that  an  air¬ 
plane  is  merely  a  collection  of  spare  parts  flying  in  close  formation, 

ft  should  be  noted  that  the  above  law,  regrettably,  cannot  be  limited  solely  to 
airborne  electronics.  For  example,  even  that  most  "ground-borne''  item  of  mili¬ 
tary  hardware,  the  tank,  is  a  notorious  offender.  The  M60A2^  the  first  tank  hav¬ 
ing  an  all-electric  turret  control  system,  contained  35,000  parts  in  the  turret  alone 
(and  in  the  field  performed  for  many  years  exactly  like  a  tank  with  35, 000  parts  in 
the  turret  alone).  It  was,  in  fact,  just  such  a  design  which  once  caused  Dr.  John 
Foster,  then  the  Director  of  Defense  Research  and  Bngineeringr  in  an  understand¬ 
able  moment  of  pique,  to  answer  a  question  as  to  how  one  might  best  defeat  a 
tank  assault  by  saying,  "Give  them  plenty  of  room  to  run  around  and  they  will  all 
break  down!"  When  considering  the  enormous  logistical  burden  created  by  such 
problems  of  unreliability,  some  solace  can  perhaps  be  derived  from  the  reali¬ 
zation  that  if  the  Soviet  Union's  tanks  have  no  better  reliability  and  repair  rates 
than  ours,  then  with  their  huge  inventory  the  Russians  are  stuck  with  more 
broken  tanks  at  any  given  moment  than  we  own  altogether. 

What,  of  course,  is  happening  is  that  as  component  reliability  improves,  more 
components  are  crammed  into  each  system  to  provide  more  and  more  capabil¬ 
ity— that  is,  more  capability  during  those  interludes  wherein  the  system  is  not 
broken.  A  modern  jetliner  has  about  4.5  million  parts,  including  100  miles  of  wir¬ 
ing.  The  Nike  Hercules  air  defense  system  contained  well  over  one  million  parts. 
But  if  a  system  has  one  million  parts,  each  with  a  reliability  of  99.9999  percent  for 
performing  some  specified  mission,,  the  overall  probability  of  the  mission  failing 
is  over  60  percent.  The  foreman  of  a  tank  plant  perceptively  explained  the  solu¬ 
tion  in  the  following  terms:  JThe  part  you  engineers  don't  put  on  the  machine 
atnrt  going  to  cause  no  trouble.," 

Thomas  Paine  summed  it  all  up  in  the  1790s  when  he  counseled,  "The  more 
simple  anything  is,  the  less  liable  it  is  to  be  disordered,  and  the  easier  repaired 
when  disordered."  Sadly,  it  has  become  commonplace  to  view  high  technology 
and  simplicity  as  contradictory  terms.  The  two  are  not,  in  fact  if  not  in  practice, 
antonyms.  The  problem  is  to  use  technology  in  a  fashion  which  engenders 
simplicity.  Who  could  argue,  for  example,  that  today's  pocket  calculators  are  less 
reliable  than  their  18,000- vacuum-tube  predecessor,  the  ENlAC,  which  complete¬ 
ly  filled  a  room? 

Law  Number  XIX,  which  states  that  expensive  systems  won't  work,  can  be 
seen  to  pose  a  particularly  serious  dilemma  to  equipment  designers  when  it  is  ap¬ 
plied  in  conjunction  with  Law  Number  X,  which  already  has  noted  that  inexpen- 
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give  systems  are  not  possible  (they  require  infinite  testing}.  This  all  may  be 
academic,  however,  since  it  has  also  been  established  (in  Law  Number  VIII)  that 
before  long  it  will  not  be  possible  to  afford  any  new  systems  anyway. 

FYI 


Wc  sure  liberated  the  hell  out  of  this  place. 

— An  American  soldier.  World  War  II 

Law  Number  XX  addresses  the  matter  of  engineers  and  managers  destroying 
the  English  language  while  trumpeting  the  worth  of  their  activities;  or,  as  the  say¬ 
ing  in  Brooklyn  goes,  "It  was  the  loudest  noise  they  ever  seen/' 

Most  major  engineering  activities  depend  on  widespread  public  understanding 
for  their  funding  or  for  their  soda]  acceptance,  if  not  both,  Yet,  in  spite  of  the 
many  examples  of  contributions  to  mankind  made  possible  through  technology, 
the  general  public  still  harbors  a  considerable  skepticism  of  the  net  benefit 
wrought  by  past  technological  advances.  As  a  result,  budgetary  and  environ¬ 
mental  limitations  abound,  and  support  For  basic  research  continues  to  erode  in 
many  quarters.  The  problem  is  exacerbated  by  the  very  language  engineers  and 
managers  use  to  communicate  their  achievements,  a  language  which  appears  to 
be  formulated  to  assure  that  no  information  might  be  transmitted— either  to  the 
public  or,  frequently,  among  themselves. 

A  former  Principal  Deputy  Under  Secretary  of  Defense  for  Research  and 
Engineering,  Gerald  Dinneen,  met  this  problem  head-on,  pointing  out  that  "we 
go  to  the  Congress  and  tell  them  that  our  VVWMCCS  has  got  to  have  a  BMEW5 
upgrade^  our  Fuzzy  Sevens  have  to  be  replaced  by  PAVE  PAWS,  we  want  to  keep 
our  PARCS  and  DEW  in  operation,  we  have  to  harden  the  NEACP,  and  we  have 
to  improve  our  MEECN  with  more  TACAMO  and  begin  planning  to  replace 
AFSATCOM  with  Triple-S — and  then  we  wonder  why  no  one  understands." 

The  extent  of  the  problem  faced  by  the  uninitiated  can  begEn  to  be  appreciated 
by  considering  the  following  excerpt  from  an  Air  Force  document  on  the  im¬ 
plementation  of  the  new  acquisition  policy,  A-l09r 

—The  HQ  U5AF/RD  sends  the  draft  MENS  through  SAF/ALR  to 
0USDKE  for  OSAF,  OSD,  DIA  and  OJCS  staff,  level  comment. 

—The  HQ  USAF/RD  OPR  develops  the  for-coordi nation  draft 
MENS  and  presents  the  MENS,  comments  and  proposed  solution 
approach  to  the  HQ  USAF  RRG  for  corporate  review  in  lieu  of  the 
underlying  SON(s). 


46 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


Augustine's  Laws 


75 


To  the  unwashed,  this  might  convey  a  message  something  like: 

—The  blank  blank/ blank  blank  sends  the  draft  blank  through 
blank/ blank  to  blank  for  blank,  blank,  blank,  and  blank  staff-level 
comment. 

—The  blank  blank/blank  blank  develops  the  for-coordination 
draft  blank  and  presents  the  blank r  comments  and  proposed  solu¬ 
tion  approach  to  the  blank  blank  blank  for  corporate  review  in  Iteu 
of  the  underlying  blank(s). 

Clearly,  having  drawn  such  a  blank  when  dealing  with  the  process  of  replac¬ 
ing  ROCs  (required  operational  capabilities)  with  MENS  (mission  element  need 
statements),  GORs  (general  operational  requirements)  and  SONs  (system  opera¬ 
tional  needs),  one  can  understand  why  there  are  those  wTho  have  been  able  to  con¬ 
clude  only  that  somehow  SON  of  MENS  must  have  been  GORd  by  ROCS. 

Of  course,  the  liberal  use  of  acronyms  and  other  means  of  obfuscation  does 
have  the  advantage  of  making  sometimes  pedestrian  material  appear  rather 
erudite  in  that  it  becomes  more  difficult  to  comprehend.  Who,  for  example, 
would  pay  a  medical  doctor  520  m  exchange  for  his  scribbling  on  a  piece  of  paper 
'Take  two  aspirin^  Hence,  the  practice  of  writing  prescriptions  in  Latin  or,  at  the 
very  least,  using  indecipherable  handwriting, 

A  practitioner  who,  rather  than  admonishing  "Take  two  aspirin/'  could 
prescribe  Take  two  acetylsalicylk  acid"  and,  in  addition,  do  so  with  po6r  pen¬ 
manship  could  very  likely  qualify  as  a  specialist  and  thereby  command  at  least 
$40  for  the  services  rendered. 

As  might  be  expected,  the  potential  of  uncommunicative  communication  has 
not  gone  unnoticed  by  the  government  and  other  large  organizations.  That  mosK 
intimidating  of  all  documents,  the  federal  income  tax  Form  1040,  is  generously 
sprinkled  with  IRAs,  HRs,  IRSs,  ILS.s,  FICAs,  RRTAs,  R&RPs,  Els,  EICs, 
ZIPs  .  ,  ,  and,  ignominiously,  something  called  "WINs," 

This  striving  to  impress  is  also  evidenced  above  the  entrances  to  public 
buildings,  where  the  inscriptions,  presumably  for  the  edification  of  tourists,  are 
of  course  offered  in  Latin.  It  thus  may  be  that  no  one  really  knows  what  E 
Pluribus  Unum  really  means  .  ,  .  but  no  one  can  question  that  it  is  impressive. 

A  few  years  ago  when  in  the  midst  of  the  national  anti-ballistic  missile  debate 
the  name  of  the  then- troubled  Zeus  missile  was  changed  to  Spartanr  it  was  only  a 
matter  of  hours  until  some  knowing  wag  had  posted  a  sign  on  a  Pentagon  bulletin 
board  proclaiming:  "Spartan:  Special  Political  Advantages  Realized  Through  Ad¬ 
vanced  Names/'  A  few  years  later,  the  oft-analyzed  but  never-deployed  ad¬ 
vanced  manned  strategic  aircraft,  AMS  A,  became  known  among  its  much- 
suffering  advocates  as  "America  s  Most  Studied  Aircraft/' 
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Similarly,  at  a  security  gate  at  Cape  Kennedy  on  the  approach  to  one  of  the 
launch  pads  is  a  sign  which,  among  a  number  of  cautions  about  explosives,  flam¬ 
mable  liquids,  falling  objects,  high  voltages,  etc.,  conspicuously  warns  visitors 
that  JJP0Vs  are  prohibited/'  Now,  this  is,  of  course,  cause  for  consternation 
among  those  who  may  wish  to  enter  but  are  somehow  not  exactly  certain  whether 
they  have  a  PQV  in  their  possession.  It  therefore  can  be  with  no  inconsiderable 
relief  that  one  learns  that  a  PQV  is  merely  govemmentese  for  "privately  owned 
vehicle,”  i.e.,  presumably  a  shortened  form  for  the  word,  "'car/'  Correspon¬ 
dingly,  a  "range  extension  system/'  better  known  as  an  RES  in  guided  bomb 
parlance,  is,  in  less  knowledgeable  circles,  merely  referred  to  in  its  short  form  as  a 
"wing"! 

There  are  those  individuals,  both  outside  the  government  and  inside,  who  are 
endowed  with  that  special  talent  to  take  fairly  straightforward  concepts  andr 
through  suitable  embellishment,  make  them  quite  nearly  incomprehensible.  The 
original  lucid  statement  of  the  acquisition  policy  which  David  Packard,  then 
Deputy  Secretary  of  Defense,  was  to  promulgate  for  the  Department  of  Defense 
was  written  by  himself  and  had  an  acronym  content  of  only  0,2  percent  of  the 
words  contained  therein.  However,  by  the  time  this  statement  was  translated  by 
acrony  mol  agi  sts  (so  it  could,  presumably,  be  more  readily  understood)  into  the 
regulation  which  underpins  much  of  the  Defense  Department's  present  acquisi¬ 
tion  policy  (DODD  5000,1),  acronyms  comprised  fully  10  times  the  above  frac¬ 
tion  of  all  the  words  in  the  document.  It  seems  doubtful,  indeed,  that  Deputy 
Secretary  Packard  would  ever  recognize  his  policy  in  its  new,  improved  form. 

It  is  suggested  that  there  are  those  who  believe  that  a  measurement  of  the  per¬ 
cent  of  words  in  a  particular  work  which  takes  the  form  of  acronyms  can  be  used 
to  determine  the  implicit  worth  of  that  work.  Clearly,  the  greater  the  number  of 
acronyms  the  greater  the  intellectual  value  of  the  material  since,  obviously,  the 
last  thing  anyone  engaged  in  communicating  would  wish  to  do  would  be  to  deny 
a  portion  of  the  audience  the  message  being  conveyed.  Thus,  in  view  of  their 
widespread  use,  acronyms  must  be  concluded  to  be  a  valuable  contributor  to  the 
worth  of  most  material. 

Figure  25  examines  this  premise  and  presents  for  a  number  of  important  acqui¬ 
sition  documents  the  actual  acronym  use-factor,,  called  the  "acronym  activity  in¬ 
dex”  (AAI),  measured  in  the  fundamental  unit  called  a  "GLOP”  (itself,  not  sur¬ 
prisingly,  an  acronym  for  "groups  of  letters  for  obfuscating  points").  The  success 
achieved  in  the  bountiful  use  of  acronyms  in  these  documents  is  evident  from  the 
enviable  ratings  shown.  These  ratings  are  particularly  creditable  when  contrasted 
with  those  of  the  more  acronymically  impoverished  examples  from  other  writings 
which  are  also  included  at  the  bottom  of  the  scale.  Clearly,  communication  in  the 
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FIGURE  75 

Use  of  Acronyms  in  Aiding  Communication 


ULTIMATE  GOAL 
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SECRETARY  OF  DEFENSE  SCHEDULE  ANNOUNCEMENT  FOR  FY  32  Bft 
PROGRAM,  PLANNING  &  BUDGETING  SYSTEM  (PPRSI-  CYCLE  131,51 

'DOD'r  AGENDA  FOR  BALLISTIC  MISSILE  DEFENSE  REVIEW  (19.7) 

,JDQDrr  CLARIFYING  MATERIAL  ON  MAJOR  SYSTEMS  ACQUISITION 
DIRECTIVE  119.0 

ADVERTISING  BROCHURE;  ELECTRONIC  WAR  FARE  SYMPOSIUM  113.0) 
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JJDOO"'  DIRECTIVE  El 37.1  ON  J"C3IW 

O0Dr'  DIRECTIVE  ESTABLISHING  DEFENSE  REGULATOR Y  SYSTEM  13  5] 


BILL  OF  RIGHTS  :ui 

DECLARATION  OF  INDEPENDENCE  101 

MAGNA  CARTA  (0) 

EM  COMMANDMENTS  IClJ 


materia]  acquisition  arena  has  risen  to  a  very  high  plateau.  In  fact,  a  newly 
prepared  government  document  has  a  list  of  10,000  official  abbreviations  to  be 
used  in  specifications  alone.  Approximately,  the  document  is  referred  to  as 
DOD  STD- 12." 

Law  Number  XX r  the  Comprehensive  Law  of  Incomprehensibility,  derived 
from  evidence  such  as  that  just  discussed,  can  be  stated; 

XX:  Profound  concepts  are  often  characterized  by  their  difficulty  of 
being  understood;  therefore,  persons  unfamiliar  with  Greek  or  Latin 
should  give  intellectual  depth  to  their  ideas  by  utilizing  acronyms  to 
a  degree  more  or  less  proportionate  with  the  lack  of  sophistication  of 
the  ideas  being  presented. 

There  are  still  further  advantages  to  acronymscal  "anonymity."  For  example, 
it  may  seem  quite  sensible  for  a  radar  designer  to  point  out  that  HF  and  UHF  are 
simply  too  low  frequencies  to  be  of  much  interest  for  target- tracking  applications. 
However,  to  state  that  "high  frequency  and  ultra-high  frequency  are  too  fpiu  fre- 
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quencies  to  be  of  much  interest  for  target- tracking  applications'"  would  suggest 
the  speaker  must  be  suffering  from  some  form  of  semantic  delirium,  Such  is  the 
advantage  of  being  obscure  dearly. 

The  current  trend  toward  ever-greater  proliferation  of  acronyms  does, 
however,  introduce  a  spectre  of  danger:  the  potential  advent  of  an  acronym  gup. 

The  Defense  Marketing  Survey  has  stated  that  in  carrying  out  its  services  it 
has  compiled  a  list  of  over  one  million  acronyms  which  are  in  common  usage  in 
defense  matters,  These  consist  principally  of  '"words"  made  up  of  five  or  fewer 
letters.  Since  the  number  of  possible  five-letter  (or  less)  acronyms  that  can  be 
formed  with  the  English  alphabet  is  no  more  than  about  14  million,  it  can  be  seen 
that  nearly  10  percent  of  all  possible  reasonable  acronyms  have  already  been  used 
up,  With  the  accelerating  use  of  such  nomenclature  r  the  day  when  the  creation  of 
new  systems  will  no  longer  be  possible  thus  may  not  be  too  distant.  This,  of 
course,  portends  ill  since  the  Soviet  Union  enjoys  a  position  of  inherent 
aeronymical  superiority  over  the  United  States  owing  to  its  possession  of  an 
alphabet  containing  32  letters.  Some  form  of  accommodation  with  China  and  its 
enormous  language  population  of  14 ,000  characters  would  therefore  appear  to  be 
prudent. 

Still  another  possible  solution  to  the  aeronymical  gap  would,  of  course,  be  to 
adopt  even  longer  and  less  pronounceable  letter  groupings— an  arena  in  which 
the  U,S.  Navy  has  been  in  the  forefront  for  some  time.  One  necessarily  wonders, 
however,  the  impact  even  today  on  an  organization's  or  individual's  self-esteem 
to  be  known  as  the  N A VH  LTH  RSCHEN,  the  NAVD1STCQMDTS,  COM- 
NAVOCEANCOM,  or  the  NAVMEDRSCHU.  On  the  other  hand,  this  identity 
might  not  appear  alt  that  unattractive  to  individuals  assigned  to  such  organiza¬ 
tions  as  ARF,  ARG„  N  EMI  SIS,  DRAG,  MORASS,,  or  AWFL5  (pronounced 
'"awful"),  but  would  represent  a  considerable  come-down  to  the  Chief  of  Naval 
Air  Training,  CNATRA,  better  known  simply  as  "Sinatra," 

Many  acronyms  do  not  mean  what  the  inexperienced  observer  might 
suspect  ,  ,  ,  ANTS,  GNATSr  DQGr  FROG,  COD,  APE,  RAT,  BAT,  RAM,  and 
CLAM  have  nothing  whatsoever  to  do  with  the  animal  kingdom  or  Noah's  Ark, 
Rather,  they  stand  for  airborne  night  television  system,  general  noise  and  tonal 
system.  Development  Objectives  Guide,  free-rocket  overground,  carrier  on-deck 
delivery^  advanced  production  engineering,  ram  air  turbine,  ballistic  aerial 


5.  Aerospace  Recovery  Facility,  Amphibious  Readiness  Group.  Naval  Ship  Missile  System 
Engineering  Center,  (Nuclear)  Design  Review  and  Approval  Group,  Modem  Ramjet  System  Syn¬ 
thesis,  Air  ForcE  Weapons  Laboratory. 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


Augustine  s  Laws 


79 


target;  reliability  and  maintainability,  and  chemical  low  altitude  missile,  respec¬ 
tively.* 

In  the  evolution  of  an  acronym,  letter  combinations  which  defy  pronuncia¬ 
tion  are  simply  reconfigured.  Thus,  National  Emergency  Airborne  Command 
Post,  NEACP,  becomes  the  "Knee  Cap  r;  the  Combat  Developments  Objective 
Guide  becomes  the  "Sea  Dog";  the  Nuclear  Weapons  Development  Guide 
becomes  the  "New  Dog";  the  airborne  laser  illuminator,  ranger,  and  tracking 
system  becomes  'Alley  Rats";  and  the  radar  target  sea  Merer  becomes  the  "Rat 
Scat:" 

The  next  anthropological  si  age  in  the  development  of  an  acronym  takes  place 
when  verbal  representations  of  a  set  of  letters  are  converted  back  into  a  written 
form,  a  stage  in  which,  inexplicably,  the  resulting  acronym  is  often  totally  dif¬ 
ferent  from  the  one  which  started  out  I  Thus,  the  fixed  special  surveillance  (radar) 
known  as  the  FSS-7  becomes,  when  rewrittenr  the  "Fuizy  Seven."  Or  the  electri¬ 
cal  unit,  the  Pica-Farad,  is  abbreviated  PF,  which,  after  phonetic  transliteration, 
is  itself  often  de-breviated  JJPuff,"  The  ultimate  state  of  maturity  of  an  acronym 
occurs  when  it  is  finally  written  in  lower  case  and  everyone  forgets  that  it  is  in 
fact  an  acronym,  such  as  "radar"  and  "laser." 

Actually,  those  working  on  aerospace  and  other  national  security  matters  can 
make  no  particular  claim  to  superiority  in  the  acronymical  arena.  Regulators  in 
all  areas  have  excelled  in  the  exploration  of  this  powerful  means  of  increasing 
confusion  among  the  masses.  Consider  the  world  of  federal  finance,  where  the 
unpronounceable  JJFNMA"  simply  becomes  a  Fannie  Mae— closely  related,  it  is 
said,  to  a  Freddie  Mac,  Still  other  mortgage  instruments  closely  parallel  in  termi¬ 
nology  some  of  the  expressions  already  discussed  pertaining  to  defense  matters, 
such  as  SAMs,  RAMs,  FLIPs,  and  ARMs,  Most  ominous  in  the  world  of  mort¬ 
gages  is  something  called  a  GRAM,  occasionally  pronounced  J,Gyp 

But  amid  all  this  confusion  is  to  be  found  redeeming  virtue:  Countless 
numbers  of  Russian  cryptanalysts  must  surely  be  fruitlessly  engaged  in  trying  to 
understand  what  American  managers  are  talking  about,7  Consider,  for  example, 
the  dilemma  of  a  Russian  cryptanalyst  confronted  with  the  task  of  reporting  to 
his  superior  a  passage  dealing  with  topics  such  as  the  computer  language:  "Jules 


6,  The  author  experienced  the  type  qf  problems  which  can  arise  from  such  double  meanings  on  the 
fin*  day  of  a  recent  tour  in  the  Pentagon.  While  faithfully  carrying  out  an  assigned  appointment 

schedule  on  Capitol  Hill  in  preparation  tor  a  forthcoming  confirmation  hearing,  the  author  /ell  it 
rather  inappropriate  that  typed  after  the  name  of  several  senators  on  his  calendar  was  the  notation 
TDLD  SOU/'  It  was  only  some  lime  laier  that  it  was  learned  that  r,OLD  SOB'1  can,  in  Washington, 
also  mean  Old  Senate  Office  Building-'1  Nonetheless,  the  ambiguity,  in  several  instances,  lingers  to 
this  very  day. 

7.  Just  as  are  many  American  managers. 
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own  version  of  the  international  algebraic  language,  seismic  intrusion  detection 
systems,  clear  air  turbulence,  multiple  independently  targetable  re-entry  vehicles, 
modular  electronic  warfare  simulators,  modular  electro -optica I  warfare  simu¬ 
lators,  high  altitude  particle  physics  experiments,  beacon-only  bombing  systems, 
Develpment  Objectives  Guides,  surface-to-air  missile  (systems),  battlefield  area 
reconnaissance  systems,  weather  observing  and  forecasting  systems,  hostile 
weapons  locating  systems,  submarine  anomaly  detection,  tactical  air- defense 
computerized  operational  simulators,  biological  aerosol  detection,  automatic  test 
equipment,  anti-radiation  missiles,  tables  of  organization  and  equipment,  mutual 
assured  destruction,  built-in  test,  inertial  navigation,  high  altitude  transmission 
experiment ,  and  directional  attack  mines." 

Such  a  report  by  a  Soviet  analyst  might  sound  something  like: 

JOVIAL  SIDS  CAT,  MIRV,  MEWS,  and  MEOWS, 

HAPPE  BOBS  DOG,  SAM,  BARCS,  WOFS,  AND  HOWLS. 

SAD  SAMS  TACOS,  BAD  MIRV  ATE, 

MIRVS  ARM  AND  TOE,  MAD  SAM  BIT  IN  HATE. 

,  .  .  DAM! 

In  summary,  simply  stated,  it  is  sagacious  to  eschew  obfuscation. 

CosfcVig  Enough  to  he  Useful 

live  within  your  income 

Even  If  you  have  to  borrow  to  do  it. 

—Josh  Billings 

Law  Number  XXI  explains  the  empirically  observed  relationship  between  the 
cost  of  an  item  and  the  quantity  of  that  item  which  is  purchased;  or,  as  tennis  pro 
Ilie  Nastase  noted  in  explaining  his  failure  to  report  the  loss  of  his  wife's  credit 
card,  "Whoever  has  it  is  spending  less  than  she  was/' 

Figure  26  shows  the  rather  unexpected  relationship  which  exists  between  the 
quantity  of  an  item  which  is  purchased  and  the  cost  per  unit  of  that  item.  It  tsseen 
that  most  articles  fall  along  a  constant  trend  line  which  encompasses  equipment 
spanning  from  the  $lO0-per-copy  LAW  antitank  rocket  to  the  multibillion-dollar 
aircraft  carrier  Nimltz.  Why  this  should  be  the  case  may  help  explain  some 
underlying  difficulties  in  the  material  acquisition  process. 

One  obvious  explanation  is  that  the  quantity  of  an  item  which  can  be  afforded 
depends  on  the  cost  of  the  item,  and  the  number  procured  is  a  simple  consequence 
of  that  one  fact.  This  seems  to  be  a  rather  unsatisfactory  interpretation,  however, 
since  it  implies  that  there  are  no  unique  requirements  for  larger  or  smaller  quanti¬ 
ties  of  various  types  of  equipment;  one  merely  buys  few  of  an  item  because  it  hap¬ 
pens  to  be  more  costly  or  many  because  it  happens  to  be  less  costly,  independent 
of  what  the  item  may  be  intended  to  do  or  of  the  need  for  that  item. 
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FIGURE  24 

Cost-Quantity  Trade-offs  in  Military  Hardware 


QUANTITY  data  approximate  .  .  .  BASE D  ON  UNCLASSIFIED  sources 


COST  IN  1 9fiQ DOLLARS 


In  this  regard  it  is  interesting  to  note  that  there  exists  a  maximum  acceptable 
unit  price,  IQ1  Q/N11  for  any  individual  item  of  equipment,,  and  this  price  depends 
only  on  the  quantity,  Nf  of  that  item  to  be  purchased.  Once  the  quantity  has  been 
determined,  the  striking  conclusion  is  that  the  cost  of  all  items  gravitates  to  this 
maximum.  Additional  capabilities  somehow  creep  into  the  hardware  until  the 
unit  cost  approximates  the  above-mentioned  value,  which  is  known  as  the 
'  threshold  of  tolerance/' 

Thus,  any  item  of  which  only  a  few  are  needed  can  [and  will)  be  allowed  to 
take  on  additional  features  until  the  cost  rises  to  the  vicinity  of  the  limit  de¬ 
scribed  Bert  Fowler,  Vice  President  of  the  MITRE  Corporation  and  former  Depu¬ 
ty  Director  for  Defense  Research  and  Engineering,  has  pointed  out  that  for  some 
reason  a  mess  table  on  a  nuclear  submarine  costs  substantially  more  than  a  mess 
table  on  a  conventional  submarine.  Similarly;  a  clock  in  a  Mercedes  Benz  costs  a 
great  deal  more  than  a  clock  in  a  Volkswagen.  So  it  goes  with  each  component 
until  the  capability  and  cost  of  the  entire  system  rise  to  the  threshold  of  intoler¬ 
ance  as  described  in  the  Law  of  Conservation  of  Input. 

&  -  M  -  &  -  QL  3 
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In  those  early  days,  however,  the  pressure  toward  more  capable  and  more 
costly  designs  was  already  at  work.  Professor  Wood  notes  in  passing  that  "the 
Curtis- Wright,  Jr.  airplane  was  designed  to  the  following  simple  specifications, 
listed  in  order  of  importance;  (1)  low  first  cost,  (2)  safety,  (3)  appearance  and  per¬ 
formance.  Professor  Wood  goes  on  to  explain  that  "the  actual  first  cost  achieved 
in  building  this  airplane  (about  $lr4£X?  retail  in  1930)  was  considered  at  the  time 
to  be  exceptionally  low,  though  the  safety  record  was  not  quite  so  satisfactory, 
and  the  sacrifice  of  performance  (cruising  speed  of  about  65  mph)  turned  out  to 
be  so  excessive  that  the  airplane  found  little  use  as  a  means  of  transportation  in 
competition  with  the  automobile/* 

The  seeds  of  increasing  expectations  were  sown  at  an  early  time. 

XXI:  The  features  incorporated  into  any  given  system  will  continue  to  in¬ 
crease  until  the  unit  cost  of  the  system  in  dollars  approximates  the 
Threshold  of  Intolerance,  which  is  defined  as  lO^/N1-2,  where  N  is 
the  quantity  of  the  item  which  is  to  be  purchased. 

This  trend  toward  higher  cost  is,  of  course,  exacerbated  by  the  fact  that  the 
exponent  in  the  denominator  above  is  greater  than  unity.  This  means  that  the 
high  unit  cost  which  is  acceptable  for  low-quantity  items  more  than  offsets  the 
volume  impact  of  high-quantity  items— so  that  a  contractor  does  slightly  more 
business  by  dealing  in  high-cost  /low-volume  materiel.  Similarly,  program 
managers  of  high-unit-cost  items  will  be  able  to  enjoy  the  status  of  directing 
larger  overall  enterprises  than  their  counterparts  dealing  in  more  economical 
systems,  albeit  procured  in  larger  volume. 

On  the  other  hand,  an  approximation  to  Law  XXI  is  that  the  quantity  of  an 
item  procured  multiplied  by  its  unit  cost  always  equals  1G3D  dollars.  This  provides 
a  convenient  method  of  determining  the  total  procurement  quantity  for  most  pro¬ 
grams. 

Over  the  years  others  have  studied  various  effects  related  to  the  one  noted 
herein.  A1  Flaxr  President  of  the  Institute  for  Defense  Analyses  and  former  Assist¬ 
ant  Secretary  of  the  Air  Force,  has  pointed  out  one  such  interesting  investigation 
described  in  the  1939  edition  of  Airplane  Design,  In  that  book,  K,  D,  Wood  ad¬ 
dressed  the  relationship  between  the  quantity  of  various  types  of  aircraft  which 
were  purchased  and  the  price  of  those  aircraft.  A  principal  difference  in  the  obser¬ 
vations  of  Professor  Wood  and  the  present  data  is,  sadly,  that  the  most  expensive 
aircraft  in  the  former  study  cost  less  than  S5,0DCH 
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All  Started  by  a  Spark 

Nearly  all  men  die  of  their  remedies,  and  not  of  their  illnesses, 

— Moliere 

General  of  the  Army  Omar  N.  Bradley  often  and  with  his  usual  perceptivity 
quoted  the  old  Signal  Corps  maxim  that  Congress  makes  a  general.,  but  only  com¬ 
munications  can  make  him  a  commander,  In  our  zeal  to  emulate  this  truism, 
however r  we  have  somehow  managed  to  place  ourselves  in  so  extreme  a  position 
that  it  has  sometimes  been  suggested  that  the  side  that  wins  the  next  war  will  be 
the  one  with  the  last  antenna  standing.  As  Bob  Everett,  President  of  the  MITRE 
Corporation,  has  warned  us  with  no  inconsiderable  amount  of  concern,  there  are 
those  who  would  have  us  believe: 

The  American  Soldier, 

His  strength  is  as  the  strength  of  ten, 

'Cause  he  has  LSI. 

LSI,  large-scale  integration  of  electronic  circuitry,  is  indeed  important.  But 
one  suspects  such  intangibles  as  courage,  motivation,  and  initiative  may  still  be 
worth  more  than  their  weight  in  silicon. 

Nonetheless,  The  S tar  reported  that  "if  past  wars  were  won  or 

lost  in  places  like  the  playing  fields  of  Eton,  future  wars  will  be  won  or  lost  on 
computer  terminals/'  The  magnitude  of  the  computer  explosion  has  been  il¬ 
lustrated  in  a  recent  session  at  M.I.T.  where  Michael  Dertavzos  noted  that  in  the 
next  few  decades  it  will  be  feasible  to  store  the  world's  knowledge  in  a  computer 
for  about  half  a  billion  dollars  per  IOC  But,  in  this  case,  an  "LOCr  is  not  the 
pedestrian  line  of  code/r  but  rather,  is  a  'Xibrary  of  Congress.''  Needless  to  say, 
this  is  a  potential  that  cannot  be  overlooked,  either. 

Such  viewpoints  do  point  to  a  trend  in  the  proliferation  of  electronics  which 
would  be  either  productive  or  counterproductive,  depending  on  how  they  are 
harnessed.  The  notion  of  computers  fighting  one  another  is  already  a  reality. 
Much  has  been  written  about  giant  data  processors  developing  codes  to  protect 
the  secrecy  of  messages  while  enemy  computers  simultaneously  seek  to  decipher 
those  codes.  Or,  on  a  smaller  scale,  there  are  today  computers  controlling 
countermeasures  devices  in  electronic  warfare  operations  and  enemy  computers 
managing  the  enemy’s  counter-countermeasures  equipment,  and  friendly  corn- 
put  rs  assigning  counter-counter-measures,  and.  ,  .  , 

Each  application  of  electronics  thus  seemingly  leads  to  still  another  in  an 
almost  endless  chain,  raising  the  danger  that  electronics  may  indeed  dominate  all 
equipment  before  it  can  itself  be  controlled.  Giant  computers  are  at  work  design  - 
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mg  their  own  offspring— the  ultimate  in  electronic  perpetuation.  The  extent  of 
this  prolific  trend  is  examined  in  Figure  27,  which  represents  the  fraction  of 
military  fighter/interceptor/attack  aircraft  weight  attributed  to  electronics.  It  has 
been  observed  that  airplanes  are  merely  a  form  of  truck  in  which  to  carry  elec¬ 
trons  around  the  sky.  Further,  the  trend  with  time  is,  unfortunately,  un¬ 
mistakable,  Extrapolating  once  again,  undauntedly,  certain  characteristics  of  that 
sole  airplane  which  was  proved  in  a  prior  law  (Law  Vlf])  to  exist  a  few  decades 
from  now  can  be  derived.  Namely,  it  will  be  made  entirely  of  electronics. 

As  dubious  as  St  may  seem,  in  order  to  sustain  the  above  well-established 
trend,  airplanes  will  eventually  have  to  be  built  using  black  boxes  in  place  of 
pilots  and  shooting  streams  of  electrons  or  photons;  this  since  there  will  be  no 
space  available  for  either  pilots  or  bullets.  In  this  space-age  airplane,  travel 
beyond  the  atmosphere  may  even  be  possible;  but  since  there  will  be  no  room  for 


FIGURE  27 

Trends  in  Avionics  Aboard  Fighter/Attack  Aircraft 


20 


LU  -P 

<! 

m 

li 

LL.  LU 


£h 


u.  4 


11 

£5 

o 

ULi 

J 

—I 

< 

iH- 

v> 


f  '  r*  A  * 

#  *■  &A 

f  'HHt  * 

*  F  1H 

*  . 

*  F 

v*>* 

* 

*  ii>iy 

r  iCMC. 

F  1M-0  A  ** 

9  49* 

A  hA  / 

k "■ j 

V 

i  1.01c  •  a 

f  i«.& 

t  99* 

4L  ^ 

/ 

a  iaA  * 

•  *** 

f  M 

■  *  *  IW 

*  r  ltM 

1940 


1050  I960  1070 

year  of  initial  operation 


I960 


56 


Defense  ARJ,  January  2015,  Vol.  22  No.  1 : 2-63 


January  2015 


j4ugush'nes  Laws 


85 


conventional  engines,  some  form  of  electrical  propulsion  will  presumably  be 
demanded  to  give  the  electron  its  due.  This  has,  clearly,  the  makings  of  science 
fiction,  but  the  trend  toward  ever- increasing  electronic  content  of  aircraft  does 
seem  to  deserve  a  skeptical  re-examination.  The  Law  of  Unrelenting  Electrifica¬ 
tion  unabashedly  predicts  that: 

XXII:  The  contribution  to  aviation  is  so  great  that  by  the  year  2015  there 
will  be  no  further  airplane  crashes.  Unfortunately,  there  will  be  no 
further  takeoffs  either:  Avionics  will  then  occupy  1 00  percent  of 
every  airplane's  weight, 

Orly  now,  with  the  establishment  of  this  law,  can  it  be  explained  what  Lord 
Kelvin,  who  did  so  much  to  advance  modern  science,  had  in  mind  when  he 
predicted  more  boldly  than  wisely  that  “aircraft  flight  is  impossible!"  All  those 
snickers  over  the  years  can  be  seen  to  have  been  undeserved;  he„  like  Calvin 
Ccolidger  was  ahead  of  his  time.  But  the  law  stated  herein  would  certainly  in¬ 
dicate  that  it  was  also  not  his  finest  hour  when  he  predicted,  "Radio  has  no 
future!"  There  can  be  little  question  that,  as  the  Chinese  proverb  states,  "It  is  dif¬ 
ficult  to  prophesize,  especially  about  the  future/' 

It  is  clear  that  when  Law  XXII  is  fully  realized,  there  will  be  no  space  or  weight 
remaining  on  combat  airplanes  to  carry  weapons  with  which  to  attack  the  target. 
But  even  this  may  not  be  altogether  inappropriate.  With  the  high  cost  of  modern 
air-to-ground  weapons,  it  may  prove  cheaper  to  simply  inn  undate  the  enemy 
with  the  avionics  pods  that  will  be  filling  most  of  the  stores-stations  anyway, 

A  related  circumstance  actually  occurred  during  World  War  I  when  the  Ger¬ 
man  Air  Force,  seeking  to  draw  fire  away  from  its  bases,  began  constructing  a 
false  airstrip  occupied  only  by  wooden  airplanes,  wooden  vehicles,  and  wooden 
buddings.  Unable  to  draw  the  attention  of  the  Royal  Air  Force,  the  Germans  con¬ 
tinued  to  expand  and  improve  upon  the  deception  until  finally,  having  spent 
nearly  as  much  money  as  would  have  been  required  to  construct  a  legitimate  air 
base,  they  abandoned  the  effort  in  frustration.  The  extent  of  frustration  was  not, 
however,  to  become  evident  until  a  few  days  later  when  a  lone  British  aircraft 
flew  down  the  main  runway  and  dropped  a  single  bomb! 

It  may  be  that  the  trend  toward  filling  all  available  space  within  an  airplane 
with  electronics  will  eventually  necessitate  a  return  to  the  early  days  of  aviation 
when  the  electronics  were  actually  trailed  on  a  line  behind  the  aircraft.  According 
to  the  1919  edition  of  U.S.  Amt y  Aircraft  Production  Facts,  “airplane  radio 
antenna  for  telegraph  work  consisted  of  about  300  feet  of  fine  braided  copper 
wire  trailing  below  and  behind  the  plane  from  a  suitable  reel  and  held  in  place  by 
a  lead  weight  of  approximately  IV*  pounds  attached  to  its  end."  Unfortunately, 
with  today  s  emphasis  on  low-altitude  military  flying,  it  is  doubtful  that  the  envi- 
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ronmental  impact  of  such  a  concept  would  be  acceptable.  Even  in  1919r  the  prac¬ 
ticability  of  such  a  scheme  suffered  some  doubt  in  that  it  was  duly  noted:  "Mr. 
McCurdy,  the  pilot,  had  to  pay  so  much  attention  to  flying  his  machine  that  he 
could  send  only  detached  letters  of  the  alphabet." 

In  fairness,  it  should  be  noted  that,  as  pointed  out  by  Dr.  George  Heilmeierr  a 
former  Director  of  the  Defense  Advanced  Research  Projects  Agency  and  current¬ 
ly  Vice  President  of  Texas  Instruments,  "[f  the  automotive  industry  had  pro¬ 
gressed  during  the  last  two  decades  at  the  same  rate  as  the  semiconductor  in¬ 
dust  ryr  a  Rolls  Roycc  would  today  cost  only  three  dollars,  and  there  would  be  no 
parking  problem  because  each  automobile  would  be  one-quarter  inch  on  a  side!" 

But,  at  the  same  timer  there  remain  those  cynics  of  the  role  of  electronics  who 
would  point  to  instances  in  the  space  program  where  had  a  human  not  been  on 
board  there  would  have  been  no  one  available  to  repair  the  failures  encountered 
in  the  life-support  system. 

There  are  also  those  who  might  irreverently  note  that  if  it  were  not  for  the 
radar  display  screens  in  cockpits,  there  would  be  no  place  to  affix  all  the  caution 
and  warning  stickers.  The  rampant  use  of  computers  is  such  that  there  are  now 
those  who  refer  to  an  airplane  and  its  associated  engines  as  "peripherals.^ 

This  trend  is  nowhere  better  represented  than  in  the  case  of  the  manned 
bomber.  The  World  War  II  B-29  contained  about  10,000  electronic  component 
parts,  the  B-47  approximately  20,000,  the  B-52  S0r000  and  the  B-SB  nearly 
100,000  ...  or  a  factor  of  two  each  generation.  But  this  rate  of  growth  has  been 
eclipsed  by  the  B-l,  which  is  packed  with  microcircuits  containing  as  many  active 
elements  on  a  single  chip  as  were  carried  in  an  entire  B -58.  Dr.  Allen  Puckett, 
Chairman  of  the  Board  of  Hughes  Aircraft  Company,  comments—not  too 
seriously— that  “the  real  miracle  of  the  Wright  brothers  flight  was  that  they  ac¬ 
complished  it  without  the  use  of  any  electronics  at  all."  He  explains,  The  only 
electrical  devices  in  the  WVrgJff  Flyer  were  the  magneto  and  the  spark  gap  in  each 
cylinder  of  the  engine/"  Today,  an  International  L-101I  contains  $4  million  of 
avion tcsr  which  was  roughly  the  worth  of  a  DC-7C  some  20  years  earlier.  In  fact, 
about  $1  million  in  1960  would  have  bought  every  microcircuit  then  in  existence. 

Not  only  have  airplanes  succumbed  to  the  electrifying  experience  of  embrac¬ 
ing  high  technology,  but  so  too  have  the  missiles  they  shoot.  The  Phoenix  missile, 
for  example,  contains  53Br000  active  circuit  elements,  contrasting  markedly  with 
its  forebear  of  a  dozen  years  earlier  which  suffered  through  its  existence  on  a  merE 
11$  active  elements.  Fortunately,  great  strides  have  been  made  in  increasing  the 
reliability  of  electronic  circuitry;  however,  correspondingly  great  discipline  must 
now  be  exercised  not  to  negate  this  gain  by  the  unbounded  introduction  of  more 
and  more  circuits. 
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Bit  by  Bit 

Wf  look  at  It  and  do  not  see  it, 

— Lao-tzu,  sixth  century  B,C, 

This  law  addresses  one  of  the  most  ethereal  substances  to  challenge  technical 
managers  in  many  years,  a  substance  that  seemingly  creeps  into  systems  to  an 
ever-increasing  extent,  even  in  instances  wherein  its  very  need  may  be  in  doubt.  It 
is  somewhat  as  Mark  Twain  has  noted,  "Banks  will  lend  you  money,  if  you  can 
prove  you  don't  need  it/* 

Considerable  strain  can  be  seen  to  be  building  within  the  acquisition  process 
as  engineers  and  managers  seek  to  produce  useful  products  while  complying  with 
the  plethora  of  laws  that  have  come  into  existence,  both  natural  and  man-made. 
Indeed,  laws,  like  regulations,  seem  to  grow  like  weeds  (See  Law  XV).  Compli¬ 
cating  the  effort  to  comply  with  all  the  regulations  is  the  often  contradictory 
guidance  given  by  official  bodies,  such  as  the  various  committees  of  Congress.  In 
fact,  in  several  recent  instances  the  Congress  has  gone  so  far  as  to  legislate  the  ini¬ 
tial  deployment  dates  for  new  systems  as  part  of  the  Appropriations  Act.  Jn  doing 
so  the  dates  are  law.  It  is  not  yet  dear  what  the  exact  liability  may  be  for 
managers  of  those  programs  should  they  fail  to  meet  the  prescribed 
dates— especially  in  instances  where  the  Congress  subsequently  cut  their 
budgets — but  it  is  clear  that  this  has  not  significantly  reduced  the  stress  within  the 
acquisition  process. 

The  dilemma  faced  by  those  involved  in  the  acquisition  process  can  be 
typified  by  the  difficulty  of  complying  with  both  Law  Number  VIII  and  the  law 
discussed  in  the  previous  section.  Law  XXII,  simultaneously.  The  first  of  these 
laws  ordains  that  the  cost  of  hardware  {e.g.,  airplanes}  increases  rapidly  with 
time.  To  comply  with  this  stringent  requirement  in  the  time  period  when  there 
will  be  no  additional  space  or  weight  left  in  an  airplane  (since  the  entire  volume 
will,  according  to  Law  XX IJ,  be  filled  with  electronics)  places  severe  demands  on 
a  designer.  Optimally,  what  is  needed  is  something  that  can  be  added  to  airplanes 
and  other  systems  which  weighs  nothing,  yet  is  very  costly,  and  violates  none  of 
the  physical  Jaws  of  the  universe,  such  as  the  law  of  gravitation  or  the  laws  of 
thermodynamics. 

This  might  appear  to  be  an  insurmountable  challenge;  however,  as  a  result  of 
(He  traditional  ingenuity  characteristic  of  system  designers,  it  can  be  reported 
with  confidence  that  such  an  ingredient  has  already  been  found. 

[t  is  called— software. 

A  principal  properly  of  software,  the  phantom  of  modern  technology,  can  be 
seen  in  Figure  23 ,  which  illustrates  the  trend  toward  ever-increasing  quantities  of 
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FIGURE  2 El 

Trends  in  Software  Growth 
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software  in  any  given  family  of  systems,6 

There  are,  in  factr  three  separate  growth  modes  evidenced  by  software.  The 
first  two  of  these  are  from  generation-to-generation  of  new  items  of  equipment 
(from  an  F-4  aircraft  to  an  F-14}  and  from  version-to-version  of  a  given  item  of 
equipment  (Titan  E  to  Titan  II  to  Titan  III).  The  third  growth  mode,  an  internal 
growth  mode,  reflects  the  increase  in  quantity  of  software  from  the  time  the  given 


ft.  The  grouping  of  the  data  shown  in  Figure  26  into  the  categories  oi  unmanned  and  manned 
systems  is  interest ingP  but  is  most  likely  a  figment  of  the  rather  modest  data  haw  available  wilh  which 
to  Lteat  I  his;  topic,  although  there  can  be  lit  Lie  doubt  of  the  reality  of  the  growth  trend  within  a  givm 
class. 
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job  is  initially  scoped  until  it  has  actually  been  completed.  This  is  often  the  most 
exasperating  mode  of  software  growth.  It  has  been  accurately  Stated  that  if  you 
automate  a  mess,  you  get  an  automated  mess.  Figure  28  addresses  the  former  two 
modes  and  suggests  a  growth  rate  on  the  order  of  a  factor  of  10  every  10  years,* 
Law  Number  XXIII r  the  Law  of  the  Piranha  Principle,,  is  derived  from 
evidence  such  as  that  shown  in  Figure  28,  with  a  strong  degree  of  the  encourage¬ 
ment  from  empirical  evidence  on  the  internal  mode  of  growth,  and  is  stated  as 
follows: 

XXI  Hr  Software  is  like  entropy.  It  is  difficult  to  grasp,  weighs  nothing,  and 
obeys  the  Second  Law  of  Thermodynamics;  he.,  it  always  increases. 

Large-scale  use  of  software  can  probably  be  traced  back  to  the  SAGE  (semi¬ 
automatic  ground  environment)  air  defense  system  of  the  late  1950s,  which  was 
implemented  using  computers  comprising  S8r0G0  vacuum  tubes  and  consuming 
L5  megawatts  of  power.  The  real-time  operating  program  for  this  computer  con¬ 
tained  about  100,000  instructions  (backed  by  support  programs  of  112  million  in¬ 
structions).  A  subsequent  ballistic  missile  defense  system.  Safeguard,  contained 
2.2  million  instructions,  of  which  755,000  were  real  time,  illustrating,  once  again, 
the  growth  propensity  inherent  to  software.  There  are  those  who  would  suggest 
that  the  contribution  of  such  degrees  of  complexity  will  be  excelled  only  by  the 
projected  advent  of  the  WQM,  the  write-on ly-memoryr 

Various  studies  have  been  conducted  which  suggest  that  over  the  last  25  years 
the  hardware /software  portions  of  the  cost  of  major  systems  are  shifting  from  an 
initial  80/20  hardware/software  ratio  to  a  ratio  approaching  20/80  in  the  decade 
ahead.  It  can  be  safely  reported  that  the  problems  encountered  in  development 
programs  have  managed  to  stay  abreast  of  this  trend. 

Actually;  software  exhibits  many  of  the  same  properties  as  hardware.  It  is 
subject  to  human  error  (typically  one  error  per  100  source  lines  of  code),  "reliabil¬ 
ity"  problems,  and  high  penalties  for  failure  to  discover  problems  early  in  the 
development  effort.  Barry  Boehm  of  TRW  has  collected  data  which  show  the  cost 
of  correcting  software  errors  at  various  points  in  a  development  activity  relative 
to  the  cost  incurred  if  the  error  is  discovered  tn  the  coding  phase.  The  cost  is  a  fac¬ 
tor  of  5  greater  when  not  discovered  until  the  acceptance  test  phase  and  a  factor 
of  15  greater  when  uncovered  an  the  operational  phase.  It  is  left  to  Weinberg's 
Second  Law  to  observe  that  if  builders  built  buildings  the  way  programmers  write 
programs,  then  the  first  woodpecker  that  came  along  would  destroy  civilization! 


1 

$.  The  author  is  indebted  to  Stephen  L.  Copp&  fo  f  his  assistance  in  cdlrcling  I  he  data  presented  in 
Fipira 
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A  classic  example  of  tire  perversity  of  software  was  encountered  in  the 
Mariner  program  when  on  the  Mariner  1  flight  the  lack  of  a  single  dash  over  a 
symbol  in  a  little-used  routine  (the  guidance  module  For  failed  doppler  radar) 
resulted  in  a  multimillion-dollar  spacecraft  striking  out  on  its  own  to  explore  the 
distant  universe  instead  of  observing  Venus  as  its  human  masters  had  intended. 
But  if  software  is  perverscr  it  is  not  without  some  redeeming  virtues.  The  next 
Mariner  flight  was  saved  when  the  same  set  of  equations  (with  the  dash  safely  in 
place)  managed  to  keep  Mariner  2  on  target  in  spite  of  an  uncontrollable  roll  in 
the  launch  vehicle  which  caused  loss  of  ground  contact  75  miles  before  full  lock 
was  re-established. 

But  if  the  state-of-the-art  in  managing  software  development  is  in  some 
respects  primitive,  the  acronym icat  language  used  to  cloud  the  art  from  those 
managers  necessarily  thrust  onto  the  periphery  of  such  activity  has  reached  a  high 
degree  of  maturity  indeed.  This  language  is  laced  with  a  veritable  core-dump  of 
bauds,  bits,  and  bytes,  MIPS,  MOPSr  and  BOPS.  In  fact,  the  highest  order  of 
acronym  teal  language  thus  far  in  use  appears  to  have  been  created  by  software 
specialists  working  on  command  and  control  systems— thus  effectively  thwarting 
all  those  senior  executives  who  may  have  had  the  audacity  to  think  it  was  their 
role  to  command,  or  perhaps  even  to  control.  But  the  unquestioned  greatest 
semantical  contribution  of  the  software  art  is  the  term  originally  coined  to 
describe  one  million  floating  point  operations  but  which  can  be  seen  herein  to 
have  much  broader  applicability  in  describing  entire  programs — or  even  entire 
groups  of  programs — he.F  the  ,,megaflop."|| 
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Years  of  process  improvement  in  defense  acquisi¬ 
tion  have  produced  many  isolated  best  practices  that 
failed  to  become  widespread  standard  practices.  The 
authors'  research  identified  six  factors  critical  to 
seeing  best  practices  adopted  as  standard  practices. 
Both  contextual  and  managerial  in  nature,  these 
address  the  extent  to  which  standardization  is  pulled, 
pushed,  and  practical.  They  organize  the  factors  in 
a  simple  2x3  framework,  explain  the  nature  of  each 
factor,  provide  examples  of  each  factor,  and  discuss 
each  factor’s  implications  for  defense  acquisition. 
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The  military  services  have  invested  heavily  in  process  improve¬ 
ment  over  the  past  several  years,  with  decidedly  mixed  results  in  the 
field  of  acquisition  (Browning  &  Sanders,  2012;  Fox,  2011;  Smith,  2003). 
While  process  improvement  efforts  yielded  impressive  gains,  too  often 
these  improvements  did  not  spread  throughout  the  defense  acquisition 
community,  remaining  isolated  best  practices  rather  than  becoming 
widespread  standard  practices. 

For  example,  in  the  authors'  experience,  several  efforts  to  reduce  acqui¬ 
sition  cycle  times  produced  impressive  breakthroughs,  often  with  cycle 
times  reduced  40-60  percent.  And  yet,  we  see  little  evidence  that  the 
efforts  producing  these  performance  gains  are  becoming  widespread 
standard  practices. 

Consider  the  perspective  of  members  of  a  defense  acquisition  program 
team  who  had  greatly  reduced  their  source-selection  time,  allowing 
a  badly  needed  system  to  be  put  under  contract  months  earlier  than 
expected.  No  one  on  the  team  could  identify  a  single  request  to  share 
ideas  with  other  source-selection  teams.  Furthermore,  members  of  the 
successful  team  were  not  confident  that  members  of  this  team  would 
apply  lessons  learned  from  their  effort,  even  to  their  own  future  source- 
selection  work! 

This  failure  to  leverage  improved  processes  in  pioneering  programs  and 
subsequently  implement  new  work  standards  across  similar  programs, 
is  greatly  limiting  the  return  on  investment  from  process  improvement 
in  acquisition.  Indeed,  such  failure  can  be  viewed  as  a  strong  causal 
factor  and  contributing  explanation  as  to  why  process  improvement 
has  failed  to  generate  the  overall  performance  gains  desired  by  the 
acquisition  community. 


Background 

In  our  research,  we  conducted  in-depth  field  studies  on  organi¬ 
zations  with  notable  successes  and  failures  at  standardizing  best 
practices  (Wicht  &  Crawley,  2012).  In  compliance  with  the  security 
requirements  of  participating  organizations,  they  will  be  referred  to 
as  Defense  Contractor,  Diversified  Corporation,  General  Hospital, 
Heavy  Equipment,  Information  Technology  (IT)  Manufacturer,  Mutual 
Insurance,  and  Structural  Fabrications. 


66 


Defense ARJ,  January  2015,  Vol.  22 No.  1 : 64-83 


January  2015 


Our  field  interviews  and  practical  observations  from  time  spent  in  these 
companies  uncovered  six  determinants  of  process  standardization  that 
we  have  organized  into  a  3x2  matrix  (Table  1).  The  matrix  captures 
three  broad  types  of  forces  we  labeled  as  pull,  practicalities,  and  push. 
"Pull”  refers  to  motivations  found  within  individuals  (Harris  &  Lewis, 
2012),  while  "push”  forces  are  those  brought  to  bear  by  factors  outside 
the  individual  (Cash,  Earl,  &  Morison,  2008;  Edison  &  Murphy,  2012; 
Roper,  2011).  "Practicalities”  deal  with  the  nature  of  the  work  and  how 
readily  it  lends  itself  to  standardization  (Cash  et  al.,  2008;  Chatterjee, 
2013;  Thomke  &  von  Hippel,  2002). 


TABLE  1.  SIX  DETERMINANTS  OF  PROCESS  STANDARDIZATION 

Three  Types  of  Forces 

Pull  Practicalities 

Push 

Two 

Origins 

Organizational 

Context 

i  ,  3.  Replicability 

1.  Inherent  Stakes  , .  ..  . 

of  Work 

5.  Organizational 
Alignment 

of 

Forces 

Managerial 

Actions 

2.  Making  Advantages  4.  Implementing 
Visible  Standard  Work 

6.  Driving 
Compliance 

We  found  each  of  these  forces  could  originate  and  grow  out  of  organi¬ 
zational  context  and/or  managerial  actions.  "Organizational  context” 
refers  to  those  factors  "built  into”  the  workplace  or  organization  inde¬ 
pendent  of  any  new  action  taken  for  the  specific  purpose  of  standardizing 
best  practices  (Chatterjee,  2013;  Szajnfarber,  Richards,  &  Weigel,  2011). 
As  we  will  discuss,  these  are  factors  such  as  the  inherent  stakes  of  the 
work,  the  replicability  of  the  work,  and  facets  of  the  organization  struc¬ 
ture.  "Managerial  actions”  refers  to  measures  taken  for  the  specific 
purpose  of  seeing  best  practices  spread  throughout  the  defense  acqui¬ 
sition  community  to  ultimately  become  standard  practices  (Garvin, 
Edmondson,  &  Gino,  2008;  Kehoe,  2010;  Pearson,  2002).  These  include 
making  the  advantages  of  standardization  more  visible,  implementation 
of  standard  work,  and  greater  emphasis  on  compliance. 

Collectively,  these  six  organizational  and  managerial  forces  have  a 
tremendous  impact  on  the  extent  to  which  organizations  are  able  to  stan¬ 
dardize  best  practices.  In  the  remainder  of  this  article,  we  look  at  these 
forces  in  detail  and  consider  their  implications  for  defense  acquisition. 
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Findings 

Contextual  Forces  Creating  Pull— Inherent  Stakes 

Several  of  the  organizations  we  studied  enjoyed  considerable  suc¬ 
cess  in  converting  isolated  best  practices  into  widely  deployed  standard 
practices,  but  none  of  them  found  it  easy.  Even  the  most  successful  could 
identify  areas  within  their  organizations  where  the  benefits  of  standard¬ 
ization  and  replication  did  not  warrant  the  costs.  Individuals  referred  to 
the  “stakes”  not  being  high  enough  to  warrant  the  effort. 

How  high  must  the  stakes  be?  In  our  research,  firms  committed  to  stan¬ 
dardizing  on  best  practices  when  it  was  literally  a  matter  of  life  or  death 
(Cash  et  al.,  2008;  Pearson,  2002).  In  other  words,  this  happened  when 
the  health  or  life  of  individuals,  or  that  of  the  organization  as  a  whole,  was 
seen  as  being  at  stake.  A  manager  at  Structural  Fabrications  (anonymous 
personal  communication,  June  2008)  explained  it  this  way: 

We  have  always  been  a  production  company.  It  is  the 
heart  of  how  we  compete  and  the  key  to  our  success. 

Selling  a  commodity,  we  compete  largely  on  cost  and 
quality,  and  if  we  don't  get  production  right,  nothing 
else  matters.  There  are  also  lots  of  ways  people  can  be 
hurt  in  our  production  areas,  so  we  are  always  working 
on  improved  safety.  Put  all  of  that  together,  and  it  just 
makes  sense  that  we  look  hard  for  every  opportunity  to 
improve  our  production  process.  We're  constantly  learn¬ 
ing  from  one  another  across  shifts  and  across  production 
areas.  The  stakes  are  just  not  as  high  in  other  parts  of 
our  business.  For  example,  in  business  development  you 
won't  find  the  same  effort  to  standardize  processes. 

Similarly,  Defense  Contractor  identified  standardization  of  its  engi¬ 
neering  processes  as  critical  to  its  survival.  The  company  had  suffered 
through  past  problems  with  inconsistent  engineering,  and  those  incon¬ 
sistencies  were  widely  seen  as  the  root  cause  in  the  company's  loss 
of  many  millions  of  dollars  and  a  damaged  reputation  in  its  industry. 
“Fixing  engineering”  came  to  be  seen  as  essential  to  the  firm's  continued 
existence,  and  several  dramatic  steps  were  taken  to  ensure  that  best 
engineering  processes  were  standardized  across  the  firm.  But  there  was 
little  evidence  of  similar  efforts  anywhere  else  in  the  company. 
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World-class  examples  of  standardization  of  best 
practices  exist  in  many  branches  of  the  mili¬ 
tary.  One  merely  has  to  look  at  where  the  stakes 
are  a  matter  of  life  or  death.  In  weapon  systems 
operation,  national  security,  and  nuclear-based 
strategic  defense,  widespread  efforts  ensure  that 
process  improvements  become  new  standard 
practices.  Operations  in  these  areas  are  subject 
to  constant  scrutiny  with  ongoing  reviews  looking 
for  better  ways  of  doing  things.  Once  better  ideas 
are  identified,  they  are  captured  as  standard  work 
and  spelled  out  in  procedures,  training,  checklists, 
and  inspections  (Cash  et  al.,  2008).  It  would  be 
almost  unthinkable  not  to  take  these  measures 
because  the  stakes  of  failing  to  do  so  are  so  obvi¬ 
ously  high. 


We  have  not  observed  the  same  phenomenon 
in  defense  acquisition.  The  situation  in  defense 
acquisition  is  similar  to  what  we  observed  in 
Mutual  Insurer,  where  we  saw  very  little  evidence 
of  systematically  sharing  best  practices  across 
operating  centers  or  sales  districts,  even  though 
the  work  done  in  each  Mutual  Insurer  location 
was  virtually  identical  to  that  done  elsewhere. 

The  most  common  answer  in  response  to  ques¬ 
tions  about  this  lack  of  standardization  was  very 
revealing  in  that  it  highlighted  the  importance  of 
perceived  high  stakes  as  a  driver:  “Standardization  across  organizational 
boundaries  is  hard.  Why  do  it  if  we  can  get  satisfactory  performance 
working  on  our  own?” 


In  summary,  the  perceived  stakes  inherent  in  defense  acquisition  are  not 
sufficiently  high  to  be  an  important  driver  of  efforts  to  standardize  and 
replicate  processes.  Note  the  emphasis  on  perceived  stakes;  the  actual 
stakes  are  really  quite  high,  suggesting  the  need  for  managers  to  make  the 
stakes  more  visible  (Edison  &  Murphy,  2012;  Kehoe,  2010;  Roper,  2011). 
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Managerial  Forces  Creating  Pull— Making  Advantages 
Visible 

In  this  research,  we  found  repeated  examples  of  the  importance  of 
an  organization  recognizing  the  advantages  that  standardizing  a  best 
practice  can  offer,  both  to  the  organization  and  to  the  organization's 
workforce  (Edison  &  Murphy,  2012;  Harris  &  Lewis,  2012;  Wicht  & 
Crawley,  2012).  Without  managerial  intervention  to  make  payoffs  more 
visible,  there  was  often  nothing  to  attract,  or  “pull,”  the  workforce  toward 
adopting  best  practices. 

At  IT  Manufacturer,  a  struggling  unit  had  come  up  with  a  radically 
different  way  of  contracting  for  reverse  logistics  services.  These  were 
outsourced  services  involving  either  reselling,  recycling,  or  scrapping 
returned  computer  equipment.  The  innovation  was  a  clear  winner,  pro¬ 
ducing  significant  financial  payoffs  in  its  first  application.  In  a  nutshell, 
it  expanded  the  conceptualization  of  reverse  logistics  to  being  a  revenue 
generator  rather  than  simply  a  cost  center. 

This  shift  in  thinking  and  in  contracting  generated  clear  wins  for  both  IT 
Manufacturer  and  its  vendor.  The  vendor  grew  revenues  and  profits,  and 
IT  Manufacturer  recognized  higher  revenues  from  new  ways  of  reselling 
returned  items.  However,  these  gains  were  not  at  all  visible  to  those  doing 
the  work  in  the  two  organizations  involved.  Instead,  the  vendor's  work¬ 
force  saw  only  that  they  were  doing  more  work  as  they  pursued  new  ways 
of  generating  revenue  from  returned  items.  Inside  IT  Manufacturer,  the 
production  division  saw  only  that  its  charges  for  reverse  logistics  went 
up  as  the  service  provider  was  paid  a  higher  processing  fee  per  returned 


While  leadership  advocated  greater  use  of  the  new 
contracting  arrangement,  employees  on  both  sides 
saw  no  advantages  and  resisted,  resulting  in  no 
movement  toward  spreading  this  better  way  of 
contracting  to  other  parts  of  the  organization . 
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item.  IT  Manufacturer  recorded  increased  revenues  and  profits,  but  the 
revenues  were  assigned  to  the  sales  division,  and  the  increased  profits 
accrued  only  at  the  firm  level. 

While  leadership  advocated  greater  use  of  the  new  contracting  arrange¬ 
ment,  employees  on  both  sides  saw  no  advantages  and  resisted,  resulting 
in  no  movement  toward  spreading  this  better  way  of  contracting  to 
other  parts  of  the  organization.  To  remedy  the  situation,  leaders  at  IT 
Manufacturer  and  its  vendor  agreed  to  participate  in  a  highly  visible 
ceremony  at  which  they  exchanged  “Big  Checks”  documenting  the  finan¬ 
cial  gains  from  the  first  year  of  using  the  new  contracting  arrangement. 
Once  employees  saw  the  amounts  on  the  checks  and  realized  the  impact 
of  the  new  contracting,  they  became  converts.  Today,  the  new  contract¬ 
ing  arrangement  is  seen  as  a  key  competitive  advantage  for  both  IT 
Manufacturer  and  its  vendor. 

At  Defense  Contractor,  engineers  resisted  the  implementation  of  stan¬ 
dardized  engineering  practices,  arguing  that  it  would  restrict  their 
creativity  and  ability  to  do  good  engineering.  It  took  a  concerted  effort  by 
leadership  to  show  examples  of  how,  in  fact,  by  adhering  to  agreed-upon 
engineering  practices,  engineers'  lives  were  simplified,  and  time  was 
freed  up  for  doing  more  and  better  engineering. 

Note  in  the  Defense  Contractor  example,  the  firm  only  benefitted  from 
standardization  after  the  individuals  came  to  see  it  was  in  their  per¬ 
sonal  best  interest  (Edison  &  Murphy,  2012;  Wicht  &  Crawley,  2012). 
The  benefits  were  not  necessarily  monetary,  and  we  observed  the 
same  thing  at  Structural  Fabrications,  as  explained  in  this  quote  from  a 
senior  operations  leader  in  the  firm  (anonymous  personal  communica¬ 
tion,  June  2008): 

Management  saw  improvements  in  key  performance 
metrics  tracked  by  the  company  just  as  soon  as  we 
started  rolling  out  the  new  production  management 
process.  The  results  were  good  enough  to  get  managers 
to  enforce  adherence  to  the  new  processes  for  awhile. 

But  as  long  as  the  guys  on  the  line  didn't  see  an  advan¬ 
tage  to  themselves,  the  only  way  they  adhered  to  the  new 
procedures  was  by  being  forced  to  do  so  by  their  super¬ 
visors.  Where  the  new  process  spread,  it  was  because 
someone  took  the  time  to  help  the  operators  see  their 
WIIFM— their  “What's  in  It  for  Me?''  It  turned  out  there 
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were  plenty  of  advantages  for  individual  workers— less 
rework,  more  predictable  work  schedules,  safer  work¬ 
places,  etc.— but  leaders  had  to  help  the  workforce  see 
them.  Once  they  did  this,  there  was  no  turning  back. 
Now  the  new  process  is  locked  in  as  the  way  we  do  things 
around  here.  But,  where  people  never  made  the  connec¬ 
tion  between  the  new  process  and  what  matters  to  them, 
implementation  eventually  became  token  and  faded. 
Company  ROI  [Return  on  Investment]  will  only  take  you 
so  far— eventually,  you  have  to  help  people  see  what  is  in 
it  for  them.  It  is  this  one -two  punch  that  gets  the  job  done. 


This  one-two  punch  is  seldom  present  in  the  defense  acqui¬ 
sition  community.  Perhaps  senior  leadership  is  generally 
aware  of  the  advantages  of  improved  acquisition  pro¬ 
cesses,  but  do  members  of  specific  program-management 
teams  or  functionals  see  personal  advantages? 

Often,  they  do  not.  For  example,  in  a  decade  of  work 
with  military  acquisition,  we  found  few 
individuals  who  could  articulate  how  they 
would  personally  benefit  from  reducing 
throughput  time  on  a  given  program.  As 
well,  few  could  clearly  show  how  they 
personally  benefitted  from  best  prac¬ 
tices  becoming  standard.  And  very 
few  individuals  in  defense  acqui¬ 
sition  felt  their  careers  would 
be  advanced  because  of  their 
adoption  of  a  best  practice  first 
developed  elsewhere. 

Contextual 
Practicalities— 
Replicability  of  Work 

Some  types  of  work  and 
organizational  structures 
lend  themselves  to  replication 
of  best  practices  more  read¬ 
ily  than  do  others  (Cash  et  al., 
2008;  Chatterjee,  2013;  Szajnfarber 
et  al.,  2011;  Thomke  &  von  Hippel,  2002). 
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Franchise  restaurants  are  a  classic  example.  When  an  individual  Subway 
sandwich  shop  discovered  that  promoting  “$5  Foot-Longs”  generated 
tremendous  volumes  and  improved  profits,  it  was  only  a  matter  of  weeks 
until  39,000  franchised  stores  followed  suit.  Consider  the  contextual 
practicalities  making  this  possible.  Each  store  offers  virtually  identical 
sandwiches  prepared  and  sold  in  virtually  identical  ways,  and  all  stores 
are  connected  with  a  strong  and  efficient  communications  network. 
These  practical  considerations  make  it  relatively  easy  for  a  franchise 
operation  such  as  Subway  to  spread  a  good  idea  across  the  organization 
quickly. 

This  should  not  imply  that  these  practical  contextual  forces  are  suf¬ 
ficient  in  themselves  to  spread  best  practices.  Mutual  Insurance  shares 
many  of  the  characteristics  inherent  in  a  franchise;  virtually  identical 
products  and  procedures  can  be  found  across  thousands  of  agents'  offices 
and  scores  of  operating  centers.  Yet,  Mutual  Insurance  has  failed  to  see 
best  practices  spread  to  become  standard  practices  for  reasons  related 
to  several  of  the  other  five  forces  in  our  model. 

Within  defense  acquisition,  we  find  very  little  standardization  across 
processes.  In  this  arena,  emphasis  is  often  placed  on  identifying  dif¬ 
ferences  between  programs  rather  than  stressing  similarities.  While 
literally  thousands  of  pages  prescribe  acquisition  procedures,  many 
programs  still  find  it  essential  to  operate  with  virtually  unlimited  use  of 
the  so-called  “county  option”  to  create  exceptions  and  new  procedures. 
To  an  outsider,  defense  acquisition  appears  to  be  like  Mutual  Insurance 
in  failing  to  capitalize  on  the  similarities  inherent  across  its  varying 
operating  units. 

Still,  it  stands  to  reason  that  the  closer  products,  users,  and  procedures 
are  virtually  identical  across  a  large  number  of  “franchise-like”  units, 
the  more  likely  processes  can  be  standardized  (Cash  et  al.,  2008).  For 
example,  the  military  services  have  been  able  to  standardize  many 
administrative  procedures  related  to  flight  operations,  including  train¬ 
ing  techniques,  “hot  wash”  after-action  reviews,  and  maintaining  pilot 
currency.  While  every  flight  is  different  in  its  details,  in  many  ways 
flights  are  similar,  and  a  flight  team's  inventing  its  own  operations 
process  based  on  the  argument  of  its  need  for  a  “county  option”  seems 
ludicrous.  Where  commonalities  exist  across  acquisition  programs,  the 
same  opportunities  exist,  but  too  many  acquisition  personnel  are  more 
interested  in  showing  how  programs  differ  than  recognizing  fundamen¬ 
tal  commonalities  (cf.  Pearson,  2002). 
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Managerial  Practicalities— Implementing  Standard  Work 

“Standard  work”  refers  to  the  means  by  which  an  organization 
defines  and  documents  its  best  practices  to  maintain  dependable  pro¬ 
cesses  (Browning  &  Sanders,  2012;  Smith,  2003).  Standard  work  spells 
out  the  currently  accepted  best  means  of  accomplishing  a  given  task  to 
the  individual  performing  the  work.  Without  standard  work,  individuals 
lack  any  practical  means  of  implementing  standardized  processes. 

In  the  organizations  getting  the  most  from  standard  work,  managers 
invest  heavily  in  its  implementation.  At  Heavy  Equipment,  hundreds  of 
formally  designated  “owners”  are  responsible  for  continuously  improving 
their  assigned  processes.  Owners  are  selected  because  of  their  experience 
and  expertise  with  a  given  process  and  their  demonstrated  commitment 
to  continuous  improvement.  Process  ownership  entails  regularly  meet- 


Process  improvement  efforts  of  the  past  decade 
have  produced  important  pockets  of  standard  work 
in  defense  acquisition,  but  these  are  not  becoming 
standard  work  across  the  larger  enterprise  like 
they  have  in  the  best  organizations  we  studied. 


ing  with  those  that  carry  out  the  process,  with  downstream  users  of  the 
process  output,  and  with  those  working  in  related  processes.  Out  of  these 
meetings,  the  process  owners  generate  improvements  that  are  captured 
in  user-friendly  source  documents,  training  materials,  and  inspection 
standards.  Efforts  related  to  improving,  documenting,  and  training  on 
standard  work  often  consume  one-third  of  a  process  owner's  time  at 
Heavy  Equipment. 

At  Defense  Contractor,  heavy  emphasis  is  placed  on  standard  work 
as  it  applies  to  engineering.  Standard  work  was  deployed  at  Defense 
Contractor  in  the  early  2000s  with  the  advent  of  computerized  tools 
to  support  the  capture  and  dispersal  of  standard  work.  Until  then, 
small-scale  attempts  at  improved  engineering  processes  had  occurred 
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in  pockets  throughout  the  organization  for  many  years.  Eventually,  a 
concerted  corporate  initiative  to  implement  standard  work  provided 
the  most  benefit  to  the  organization.  Senior  leadership  not  only  directed 
personnel  toward  the  use  of  standard  work,  they  demanded  it.  The  most 
senior  leaders  at  Defense  Contractor  ordered  their  engineers  to  engage 
in  the  standard  work  by  insisting:  “Put  your  pencils  down,  and  don't 
continue  until  you  create  and  use  engineering  standard  work."  One 
employee  recalled  the  sentiment  and  conversation  (anonymous  personal 
communication,  June  2008)  of  that  time  as  follows: 

We  can't  operate  like  this  anymore.  This  is  a  call  to  arms. 

We're  gonna  stop,  we're  gonna  put  people  on  reducing 
our  cost  of  poor  quality  and  understanding  what's  driv¬ 
ing  that,  and  we're  not  going  to  allow  anybody  to  start 
designing  until  we  get  our  standard  work  nailed  down. 

The  results  were  dramatic;  engineering  issues,  both  trivial  and  more 
substantive,  dropped  from  thousands  a  year  to  dozens  a  year.  Today, 
senior  leadership  feels  that  engineering  standard  work  is  essential  to 
the  firm's  success. 

Process  improvement  efforts  of  the  past  decade  have  produced  impor¬ 
tant  pockets  of  standard  work  in  defense  acquisition,  but  these  are  not 
becoming  standard  work  across  the  larger  enterprise  like  they  have  in 
the  best  organizations  we  studied.  The  opportunity  is  there  for  defense 
acquisition  to  take  standard  work  to  the  next  level  and  see  isolated  best 
practices  become  organizational  standards. 

Contextual  Forces  Contributing  to  Push— Organizational 
Alignment 

Of  the  organizations  we  studied,  those  most  successful  with  stan¬ 
dardization  of  best  practices  went  to  great  lengths  to  “bake  it  in"  to 
their  larger  strategy  and  structure  (Cash  et  al.,  2008;  Chatterjee,  2013; 
Szajnfarber  et  al.,  2011).  Strategy,  structure,  and  standardization  were 
all  consciously  aligned  and  reinforced  one  another.  One  of  the  most 
powerful  types  of  alignment  we  observed  was  that  between  line  and 
staff  organizations. 

For  example,  at  Structural  Fabrications,  production  improvements 
were  priorities,  each  backed  up  with  a  centralized,  company-wide,  staff- 
supported  initiative.  These  included  initiatives  such  as  those  to  improve 
safety,  reduce  waste,  increase  employee  engagement,  etc.  Staff  groups 
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variously  described  as  “Centers  of  Excellence,”  “brain  trusts,”  “corporate 
ninjas,”  or  “subject  matter  experts”  supported  each  corporate  initiative. 
The  primary  role  of  these  groups  was  to  identify  best  practices  and  assist 
plants  in  deploying  them. 

These  staff  groups  operated  with  a  scorecard,  tracking  success  in  using 
their  expertise  to  help  the  line  organization  improve  its  performance.  For 
example,  the  group  responsible  for  driving  best  practices  in  reduction  of 
waste  tracked  operating  cost  reductions  due  to  reduced  scrap,  improved 
yields,  and  lowered  inventories,  etc.,  as  key  elements  of  its  scorecard. 
This  was  in  alignment  with  priorities  in  the  line  organization,  where 
each  plant  was  evaluated  on  overall  performance  metrics  that  could  be 
improved  by  deploying  the  proven  solutions  available  from  the  waste 
reduction  group  and  other  staff  support  groups.  This  arrangement  is 
reflected  in  Figure  1. 


FIGURE  1.  A  “MARKETPLACE”  OF  PERFORMANCE 
IMPROVEMENT  OPTIONS 


Line  Organization: 
Plants 


B 


C 


Scorecards 


Centers  of 
Excellence 
Scorecards 


In  this  simplified  and  hypothetical  depiction,  plants  are  each  respon¬ 
sible  for  delivering  gains  in  specific  improvement  targets  captured  in 
a  scorecard— say  4%  reduction  in  cost  per  unit,  6%  reduction  in  inven¬ 
tory,  5%  increase  in  production  volumes,  etc.  Meanwhile,  the  Centers  of 
Excellence  are  held  accountable  for  their  own,  initiative-specific  targets 
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that  could  be  reached  only  if  their  expertise  is  successfully  employed  by 
the  various  plants— say  $10  million  of  cost  reduction  through  reduced 
scrap,  5%  improvement  in  company-wide  plant  uptime,  or  3%  reduction 
in  corporate-wide  days  lost  to  accidents.  To  hit  their  annual  perfor¬ 
mance  targets,  the  plants  depend  on  the  expertise  resident  in  the  support 
groups.  Conversely,  to  hit  their  performance  targets,  the  support  staff 
requires  application  of  best  practices  in  the  plants  in  order  to  generate 
real  dollar  impacts. 

Corporate  leadership  did  not  seek  to  control  or  specify  which  plants 
employed  which  initiatives.  Rather,  they  created  a  system  that  encour¬ 
aged  local  leaders  to  sort  out  where  their  greatest  gains  could  be  found. 
Plant  managers  had  targets  to  hit,  and  they  had  help  to  draw  upon  in  hit¬ 
ting  these  targets,  but  which  help  they  chose  to  employ  was  left  largely 
to  them.  Meanwhile,  Centers  of  Excellence  were  accountable  for  having 
a  given  cumulative  impact  on  plant  operations,  but  there  was  no  blanket 
expectation  that  every  plant  would  employ  the  same  blend  of  initiatives 
in  hitting  its  targets.  So  in  the  hypothetical  example  we  have  here,  Plant 
B  is  relying  heavily  upon  Initiatives  1  and  2  to  deliver  its  performance 
gains,  and  Center  of  Excellence  3  was  seeing  its  greatest  impact  in  Plants 
C  and  D. 

In  our  research,  we  came  to  describe  this  arrangement  as  a  free-market 
approach  to  fostering  standardization  of  best  practices.  Rather  than  cen¬ 
trally  dictating  solutions,  the  organization  identified  several  potential 
areas  of  improvement  and  invested  resources  in  developing  centralized 
expertise  in  these  areas.  The  decision  on  how  they  could  best  engage 
with  one  another  to  hit  their  complementary  performance  targets  was 
then  left  up  to  local  leaders  in  the  plants  and  the  Centers  of  Excellence. 

We  saw  very  similar  line-staff  alignment  in  Defense  Contractor, 
Diversified  Corporation,  and  Heavy  Equipment,  and  in  every  case,  man¬ 
agers  were  quick  to  point  to  this  line-staff  alignment  as  an  important 
driver.  In  our  work  in  defense  acquisition,  we  did  not  encounter  such 
alignment.  Instead,  we  observed  much  greater  emphasis  placed  on 
program  management  (the  line  organization  in  this  case)  than  on  func¬ 
tionals  (the  staff  organization).  Certainly,  holding  program  managers 
accountable  for  cost,  schedule,  and  performance  is  appropriate.  But, 
this  is  only  the  starting  point  if  the  goal  is  to  see  best  practices  grow  into 
standard  practices.  In  most  cases,  the  functionals,  such  as  contracting, 
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financial  management,  engineering,  etc.,  "own”  the  processes.  The  pro¬ 
gram  managers  simply  employ  that  process  in  execution  of  a  single 
program,  but  the  functionals  see  their  processes  used  over  and  over. 

Given  these  realities,  many  of  the  greatest  opportunities  to  motivate 
process  standardization  rest  with  the  functionals.  It  does  make  sense 
to  evaluate  a  single  program  on  its  cost,  schedule,  and  performance. 
But,  doesn't  it  also  make  sense  to  evaluate  a  functional  on  the  aggre¬ 
gated  performance  of  all  programs  using  its  processes?  For  example, 
if  a  single  medium-size,  sole-source  acquisition  program  takes  2  years 
to  place  under  contract,  that  is  an  indictment  of  program  management 
for  that  specific  program.  But,  if  the  average  time  required  for  the  last 
50  medium-size,  sole-source  contracts  to  be  executed  is  2  years,  this 
implies  there  is  a  systemic  process  issue  that  should  be  addressed  by  the 
functional  process  owners  involved.  For  the  most  part,  functionals  are 
not  under  nearly  as  much  pressure  as  program  managers  when  it  comes 
to  improved  cost,  schedule,  and  performance,  and  this  misalignment 
appears  to  be  an  important  cause  of  limited  success  in  seeing  processes 
standardized  on  best  practices. 

Managerial  Forces  Creating  Push— Driving  Compliance 

Of  the  organizations  we  studied,  those  successful  in  standardizing 
best  practices  all  had  leaders  who  relentlessly  pushed  compliance  to 
make  it  happen  (Edison  &  Murphy,  2012;  Garvin  et  al.,  2008;  Roper, 
2011).  We  observed  two  strategies  for  driving  compliance— one  bureau¬ 
cratic,  and  the  other  behavioral. 

In  most  organizations  we  studied,  bureaucratic  controls  were  clearly 
used  to  ensure  compliance  to  standardized  best  practices  (Cash  et  al., 
2008;  Chatterjee,  2013;  Edison  &  Murphy,  2012).  Sometimes,  these 
controls  might  be  used  for  specific  functions  and/or  organizational 
levels.  For  example,  at  Defense  Contractor,  each  engineer  was  approved 
only  for  actions  specified  in  four  different  levels  ranging  from  Level  1 
(execute  standard  work  under  a  supervisor)  to  Level  4  (authority  to  set 
new  engineering  standards  and  approve  deviation  from  them).  Other 
times,  the  bureaucratic  controls  were  applied  in  a  way  that  cut  across 
functions  and  levels.  For  example,  the  product  development  process  at 
Heavy  Equipment  entails  a  tightly  controlled  set  of  hurdles  with  stan¬ 
dard  work  informing  the  appropriate  next  steps  at  every  turn.  Changes 
in  this  process  must  receive  approval  at  levels  as  high  as  the  senior  vice 
president  in  order  to  ensure  thought  has  been  given  to  potential  cross¬ 
organization  ripple  effects. 
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In  most  cases,  the  functionals,  such  as  contracting, 
financial  management,  engineering,  etc.,  "own”  the 
processes.  The  program  managers  simply  employ 
that  process  in  execution  of  a  single  program,  but 
the  functionals  see  their  processes  used  over  and 
over. 


In  the  best  organizations  we  studied,  these  examples  of  bureaucratic 
control  often  complemented  the  use  of  what  we  have  termed  behavioral 
controls.  While  bureaucratic  controls  rely  on  explicitly  codified  orga¬ 
nizational  rules  developed  and  enforced  by  management,  behavioral 
controls  entail  unwritten  norms  enforced  by  a  broader  range  of  organi¬ 
zational  members.  For  example,  managers  at  General  Hospital  found  it 
very  difficult  to  dictate  standard  processes  to  the  doctors  using  its  oper¬ 
ating  rooms.  The  operating  room  is  considered  the  sanctum  sanctorum 
of  healthcare— the  place  where  only  doctors  decide  how  medicine  will 
be  practiced  and  managers  are  held  outside.  However,  many  operating 
procedures  are  replicated  hundreds  of  times  each  day,  and  it  is  entirely 
reasonable  for  hospital  management  to  clearly  identify  any  procedures 
that  consistently  work  better  than  alternative  procedures  surgeons  may 
persist  in  using  as  a  matter  of  personal  preference. 

In  a  situation  like  this,  General  Hospital  found  it  very  useful  to  employ 
behavioral  controls  to  drive  compliance  through  peer  pressure.  They 
simply  posted  data  comparing  different  orthopedic  surgery  groups  on 
the  same  operation  in  the  doctors'  scrub  room,  without  revealing  the 
identities  of  the  different  surgery  groups.  For  example,  they  posted  the 
average  costs  and  typical  percentages  of  cases  with  complications  for 
the  seven  surgery  groups  putting  in  artificial  knees,  simply  listing  the 
surgery  groups  as  Group  A  through  Group  G.  The  data  revealed  that  the 
best  group  was  40  percent  less  expensive  than  the  worst  group  and  had 
30  percent  fewer  postoperative  complications.  Naturally,  each  surgery 
group  wanted  to  know  which  line  of  data  on  the  table  referred  to  their 
particular  practices.  And  naturally,  surgeons  tried  to  figure  out  who  was 
doing  the  best  and  worst.  When  the  worst  performing  group  saw  their 
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data,  and  recognized  that  all  the  other  surgeons  were  also  seeing  the 
same  data,  they  quickly  adjusted  their  procedures  to  bring  them  more 
in  line  with  best  practices. 

Logically,  the  strongest  levels  of  compliance  rely  on  both  bureaucratic 
and  behavioral  controls.  Consider  this  quote  about  military  flight  opera¬ 
tions  (anonymous  personal  communication,  June  2008): 

It  is  drilled  in  throughout  your  career  that  flight  opera¬ 
tions  must  take  place  by  the  book.  There  are  endless 
check  offs  where  someone  must  sign  before  a  particular 
action  is  allowed  to  take  place.  But,  just  as  important, 
there  is  a  culture  here  that  is  constantly  reinforced  by 
leadership.  As  a  result,  even  if  General  Buck  Rogers  tries 
to  climb  into  a  cockpit  without  the  right  documentation 
showing  he  is  checked  out  for  that  aircraft,  Airman  Able 
will  step  up  to  stop  him.  And  rightfully  so— we  all  count 
on  one  another  to  police  this. 

Such  examples  make  it  clear  that  military  organizations  know  how  to 
combine  bureaucratic  and  behavioral  controls  to  drive  adherence  to 
standardized  processes.  Unfortunately,  similar  examples  in  defense 
acquisition  are  hard  to  find. 


Conclusions 

We  have  identified  six  forces  that  work  collectively  to  influence 
the  extent  to  which  organizations  are  able  to  turn  their  isolated  best 
practices  into  widespread  standard  practices.  We  have  also  shown  how 
typical  defense  acquisition  organizations  are  often  deficient  in  each 
of  these.  Opportunities  exist  throughout  the  defense  acquisition  com¬ 
munity  for  dramatically  increasing  the  payoff  to  process  improvement 
efforts  in  defense  acquisition  by  isolated  innovations  becoming  common 
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practices.  But,  this  will  require  a  broad  perspective  on  the  program,  and 
a  willingness  to  engage  in  systemic  change  on  a  number  of  fronts.  Our 
hope  is  that  this  article  provides  insights  into  the  challenges  faced. 
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Cost  growth  is  a  persistent  adversary  to  efficient  budgeting  in  the  Department 
of  Defense.  Despite  myriad  studies  to  uncover  causes  of  this  cost  growth, 
few  of  the  proposed  remedies  have  made  a  meaningful  impact.  A  key  reason 
may  be  that  DoD  cost  estimates  are  formulated  using  the  highly  unrealistic 
assumption  that  a  program’s  current  baseline  characteristics  will  not  change 
in  the  future.  Using  a  weather  forecasting  analogy,  the  authors  demonstrate 
how  a  statistical  approach  may  be  used  to  account  for  these  inevitable  baseline 


changes  and  identify  related  cost  growth  trends.  These  trends  are  then  used 
to  reduce  the  error  in  initial  acquisition  cost  estimates  by  over  one  third  for 
major  defense  acquisition  programs,  representing  a  more  efficient  allocation 
of  $6  billion  annually. 
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The  Not-So-Perfect  Storm 

Inaccurate  cost  estimates  have  long  plagued  Department  of  Defense 
(DoD)  acquisition  efforts.  Despite  the  myriad  acquisition  reforms,  and 
abundant  detailed  guidance  on  cost  estimating  best  practices,  accurately 
predicting  the  eventual  cost  of  a  weapon  system  remains  difficult.  A 
Government  Accountability  Office  (GAO)  study  of  all  96  active  major 
defense  acquisition  programs  (MDAP)  in  2011  showed  a  total  cost 
increase  of  over  $74  billion  in  that  year  alone  (GAO,  2012a)— an  amount 
that  would  have  paid  for  the  2013  defense  sequestration  cuts  nearly  twice 
over.  The  total  MDAP  portfolio  cost  continued  to  grow  into  2013,  despite 
a  trend  of  reduction  in  the  number  of  programs  (GAO,  2014).  A  RAND 
study  of  completed  major  acquisition  programs  showed  that  the  aver¬ 
age  cost  estimate  error  measured  from  Milestone  B  is  about  65  percent 
(Arena,  Leonard,  Murray,  &  Younossi,  2006a).  This  figure  is  an  average 
of  overestimates  and  underestimates;  the  absolute  error  is  even  higher. 
While  researchers  and  practitioners  may  disagree  on  the  efficacy  of 
recent  acquisition  reforms  upon  improving  cost  estimates,  clearly,  there 
is  ample  room  for  improvement. 

Perhaps  the  problem  does  not  lie  with  the  accuracy  of  the  cost  estimates, 
but  with  the  fact  that  these  estimates  are  accurately  estimating  the 
wrong  thing.  For  example,  when  the  RAND  study  corrected  the  cost  data 
for  changes  in  procurement  quantity,  the  average  cost  errors  dropped  by 
over  20  percent  (Arena  et  al.,  2006a),  and  the  GAO  (2012a)  study  attrib¬ 
uted  nearly  40  percent  of  the  $74  billion  increase  to  quantity  changes.  If 
we  expect  accurate  estimates  of  the  final  cost  of  acquisition  programs, 
then  we  must  take  into  account  the  uncertainty  associated  with  program 
baselines  upon  which  these  estimates  are  based.  We  propose  a  method 
for  correcting  initial  acquisition  cost  estimates  using  observed  baseline 
deviations  from  similar  past  programs,  thus  reducing  the  average  cost 
growth  over  these  early  estimates. 

The  Defense  Acquisition  University  (DAU)  defines  cost  growth  as  “the 
net  change  of  an  estimated  or  actual  amount  over  a  base  figure  pre¬ 
viously  established.”1  Many  studies  cite  changes  to  the  Acquisition 
Program  Baseline  (APB)  as  among  the  most  significant  sources  of 
cost  growth  (Arena  et  al.,  2006a;  Drezner,  Jarvaise,  &  Hess,  1993; 
GAO,  2012a).  These  studies  often  correct  the  cost  estimates  for  these 
changes  in  an  attempt  to  determine  the  programmatic  causes  for  the 
cost  overruns.  In  this  way,  researchers  “maintain  the  integrity  of  the 
baseline”  (Drezner  et  al.,  1993,  p.  11).  These  baseline-corrected  analyses 
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A  more  accurate  prediction  of  the  eventual  cost 
of  an  acquisition  program  provides  a  better 
assessment  of  that  program’s  affordability,  thus 
better  informing  affordability  decisions. 


are  useful  for  driving  acquisition  reform,  but  they  are  less  useful  for 
informing  resource  allocation  and  affordability  assessments,  which 
are  inherently  more  concerned  with  accurate  prediction  of  actual 
program  expenditures. 


Will  Cost,  Should  Cost,  and  Real  Life 

In  a  2011  memorandum  from  the  Assistant  Secretary  of  the  Air  Force, 
Financial  Management  and  Comptroller,  and  the  Air  Force  Acquisition 
Executive  (Department  of  the  Air  Force,  2011),  the  Air  Force  established 
the  practice  of  generating  two  different  cost  estimates  dubbed  Will  Cost 
and  Should  Cost.  The  Should  Cost  estimate  is  “based  on  realistic  tech¬ 
nical  and  schedule  baselines  and  assumes  success-oriented  outcomes." 
In  contrast,  the  Will  Cost  estimate  is  based  on  an  independent  estimate 
that  “aims  to  provide  sufficient  resources  to  execute  the  program  under 
normal  conditions”  (Department  of  the  Air  Force,  2011,  p.  4).  This  notion 
that  a  program  may  cost  something  more  than  it  should  cost  implicitly 
acknowledges  that  things  don't  always  go  as  desired.  Also,  this  concept 
sets  the  precedent  that  allowances  may  be  made  for  difficulties  through 
cost-estimating  relationships  that  reference  past  development  and  pro¬ 
duction  efforts  as  a  benchmark. 

In  actuality,  the  Should  Cost  estimate  does  not  incorporate  enough  real¬ 
ism.  For  example,  common  sources  of  cost  growth,  such  as  procurement 
quantity  changes,  are  not  included  in  the  Should  Cost  estimate  since  this 
estimate  is  still  based  on  the  APB.  This  baseline  specifies  parameters 
such  as  procurement  quantity,  performance  characteristics,  program 
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duration,  and  so  on.  However,  these  baselines  almost  never  remain  con¬ 
stant  (Drezner  &  Krop,  1997),  leading  inevitably  to  changes  in  program 
cost  and  crippling  early  estimating  efforts. 

A  more  accurate  prediction  of  the  eventual  cost  of  an  acquisition  program 
provides  a  better  assessment  of  that  program's  affordability,  thus  better 
informing  affordability  decisions.  Therefore,  the  DoD  needs  a  method 
for  accurately  estimating  the  final  cost  of  an  acquisition  effort  without 
relying  on  a  fixed  baseline.  In  this  research,  we  have  developed  a  novel 
method  to  correct  early  program  cost  estimates  using  high-level  descrip¬ 
tive  programmatic  parameters.  Advanced  regression  techniques  establish 
a  relationship  between  these  parameters  and  the  cost  estimate  error  of 
past  programs,  and  then  use  this  relationship  to  predict  estimate  error 
in  similar  future  programs.  This  method  is  dubbed  “macro-stochastic” 
estimation  (Ryan,  Schubert  Kabban,  Jacques,  &  Ritschel,  2013,  p.  3). 

The  National  Oceanic  and  Atmospheric  Administration  (NOAA)  uses  a 
similar  technique  in  the  forecasting  of  hurricanes,  a  domain  that  has  seen 
prediction  accuracy  triple  in  the  last  two  decades  (Silver,  2012).  This  fact 
is  intriguing,  because  the  challenges  associated  with  predicting  the  path 
of  a  hurricane  are  remarkably  similar  to  those  of  trying  to  predict  and 
budget  for  the  cost  trajectory  of  a  DoD  program.  In  both  cases,  an  extraor¬ 
dinary  number  of  discrete,  nonlinear  elements  all  interact  in  exceedingly 
complex  ways,  serving  to  greatly  complicate  the  task  of  predicting  overall 
system  behavior.  And  while  the  two  phenomena  both  present  similar  esti¬ 
mating  challenges,  the  modeling  approaches  and  reporting  conventions 
vary  significantly. 


We  Know  What  a  Bad  Prediction 
Looks  Like 

For  a  moment,  imagine  that  meteorologists  forecast  hurricanes  in  the 
same  manner  that  the  DoD  budgets  for  acquisition  programs.  The  local 
news  channel  reports  that  a  hurricane  has  formed  in  the  Caribbean.  An 
expert  team  of  meteorologists  carefully  examines  the  key  characteristics 
of  this  newly  formed  hurricane,  including  its  current  location,  size,  speed, 
and  heading.  Based  on  this  information,  the  meteorologists  then  officially 
announce  their  prediction  for  the  hurricane:  it  will  be  a  Category  2  hur¬ 
ricane  that  makes  landfall  at  the  intersection  of  Main  Street  and  Third 
Avenue  in  Corpus  Christi,  Texas.  The  residents  of  Corpus  Christi  are 
notified  of  the  threat.  But,  24  hours  later,  the  meteorologists  follow  this 
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same  process,  and  provide  an  equally  detailed— but  vastly  different- 
prediction.  The  Day  2  prediction  is  updated  to  take  into  account  a  new 
trajectory  and  larger  size;  now  the  storm  is  predicted  to  make  landfall  at 
the  Northeast  corner  of  the  Walmart  store  in  Cameron,  Louisiana,  as  a 
Category  3  hurricane.  The  next  day,  this  process  repeats,  predicting  an 
even  larger  hurricane  with  a  new  landfall  point  in  the  parking  lot  of  the 
Spinnaker  Beach  Club  in  Panama  City,  Florida.  These  volatile  predic¬ 
tions  are  depicted  in  Figure  1. 


You  might  reasonably  have  many  concerns  about  these  estimates.  For 
example,  how  likely  is  it  that  the  hurricane  will  actually  make  landfall 
at  these  precise  locations?  You  might  wonder  why  each  estimate  only 
considers  the  current  state  of  the  hurricane  as  opposed  to  how  it  might 
change  over  time.  And,  of  course,  you  might  be  highly  skeptical  of  any 
set  of  estimates  that  varies  so  widely.  But,  this  scenario  does  have  some 
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unfortunate  similarities  with  the  DoD  cost-estimating  and  budgeting 
processes.  Although  cost  estimators  carefully  account  for  uncertainty 
in  their  cost  estimates  (based  on  a  fixed  APB),  the  official  prediction 
is  recorded  into  the  budget  as  a  point  estimate.  Their  cost  estimates 
typically  include  no  consideration  for  a  change  in  trajectory,  and  no 
indication  of  uncertainty  in  the  eventual  budget  request.  Just  like  in  our 
fictitious  forecasting  scenario,  we  have  an  early  prediction,  but  it  is  not 
a  very  good  one  since  it  is  almost  guaranteed  to  change.  Updating  the 
absurdly  specific  budget  request  at  each  milestone  is  not  an  adequate 
solution  for  addressing  this  change  since  substantial  resources  will 
have  already  been  committed  according  to  the  original  baseline.  In  fact, 
a  common  engineering  adage  presumes  that  75  percent  of  the  design 
cost  is  committed  in  the  first  25  percent  of  the  life  cycle  (Blanchard  & 
Fabrycky,  2011). 

Of  course,  this  is  not  the  way  meteorologists  forecast  hurricanes.  NOAA 
uses  supercomputers  running  millions  of  advanced  physics  simulations 
to  calculate  the  outcomes  of  minor  changes  in  the  weather's  initial 
conditions,  and  these  outcomes  are  combined  to  form  a  probabilistic 
prediction  (e.g.,  “There  is  a  10  percent  chance  of  rain  today")-  These 
simulations  are  supervised  by  experienced  meteorologists,  using  their 
knowledge  of  past  weather  patterns  to  improve  forecast  accuracy  by  up 
to  25  percent  over  computer  simulation  alone  (Silver,  2012).  This  mar¬ 
riage  of  cold  calculations  and  “squishy"  probabilistic  judgments  carries 
over  to  hurricane  prediction;  to  predict  the  storm's  path,  NOAA  uses  this 
method  of  human-mediated  simulation  (Ferro,  2013). 

But  for  the  prediction  of  hurricane  strength,  forecasters  turn  to  what  is 
essentially  macro-stochastic  estimation.  They  “compare  basic  informa¬ 
tion  from  the  current  storm,  like  location  and  time  of  year,  to  historic 
storm  behavior,"  and  use  this  information  to  predict  the  storm's  strength 
(Ferro,  2013).  In  other  words,  top-level  descriptive  parameters  are  used 
to  associate  this  storm  with  previous  storms.  The  implicit  assumption 
is  that  the  current  hurricane  will  perform  similar  to  past  hurricanes,  as 
long  as  the  right  descriptive  parameters  are  chosen.  This  combination  of 
detailed  simulation,  coupled  with  statistical  techniques  (not  to  mention 
a  healthy  respect  for  uncertainty)  produces  the  most  useful  estimate 
for  informing  evacuation  decisions.  That  is,  it  results  in  a  reasonably 
accurate  prediction  as  early  as  possible. 
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However,  embracing  uncertainty  is  not  synonymous  with  imprecision; 
for  a  prediction  to  be  useful,  it  must  not  be  overly  vague.  Most  people  are 
acquainted  with  the  graphic  that  weather  forecasters  use  to  illustrate 
the  expected  path  of  hurricanes;  an  example  is  shown  in  Figure  2.  This 
familiar  visual  form  of  prediction  has  two  important  elements: 

1.  The  Cone:  the  region  of  uncertainty  that  shrinks  as  the 
storm  approaches  land  and  provides  an  idea  of  the  confi¬ 
dence  in  the  estimate. 

2.  The  Curve:  the  change  in  trajectory  that  indicates  the  pre¬ 
dicted  path  the  storm  will  take. 


Note.  Adapted  from  Cost  Estimating  and  Assessment  Guide  (GAO-09-3SP),  by 
Government  Accountability  Office,  2009,  Washington,  DC. 
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The  Cone 

The  entire  body  of  recent  DoD  cost-estimating  guidance  empha¬ 
sizes  the  importance  of  risk  analysis,  sensitivity  analysis,  and  the 
reporting  of  confidence  in  the  program  cost  estimates  (GAO,  2009;  U.S. 
Air  Force,  2007). 1  In  fact,  one  might  admire  the  similarity  between 
NOAA's  hurricane-tracking  chart  and  a  notional  graphic  from  the  GAO 
Cost  Estimating  and  Assessment  Guide  (Figure  3)  that  illustrates  the 
trajectory  of  a  cost  estimate  baseline,  with  its  accompanying  cone  of 
uncertainty  (GAO,  2009).  Unfortunately,  the  complex  DoD  process  for 
turning  an  estimate  into  a  budget  does  not  possess  a  mechanism  for 
incorporating  uncertainty.  Despite  the  best  efforts  of  cost  analysts  to 
inform  their  customers  of  the  confidence  and  possible  risk  in  their  cal¬ 
culations,  these  warnings  are  often  interpreted  as  being  too  vague— a 
sentiment  once  expressed  by  an  irate  Harry  S.  Truman,  who  famously 
declared:  “Give  me  a  one-handed  economist!  All  my  economists  say,  'on 
the  one  hand,  on  the  other'  ”  (Krugman,  2003).  Incorporating  uncertainty 
in  budgeting  activities  requires  a  transformation  in  the  way  we  think 
about  resource  planning.  The  first  step  in  catalyzing  such  a  revolution 
is  likely  to  make  provisions  (or  mandates)  for  reporting  cost  estimate 


FIGURE  3.  CONCEPTUALIZATION  OF  COST  ESTIMATE 


TRAJECTORY  AND  CERTAINTY  OVER  TIME 
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uncertainty  and  confidence  in  acquisition  status  reports.1  However, 
acquisition  reform  is  beyond  the  scope  of  this  study.  Instead,  we  will 
focus  primarily  on  “The  Curve.” 


The  Curve 

It  is  not  always  reasonable  to  expect  that  the  DoD  can  acquire  a 
new  weapon  system  for  the  Milestone  B  “sticker  price.”  As  one  author 
recently  noted,  “Cost  Discovery  might  be  a  better  term  for  the  process 
of  updating  estimates,  because  in  retrospect  it  was  clearly  impossible 
to  produce  the  stated  capabilities  for  the  original  price”  (Cancian,  2010, 
p.  396).  It  is  rational  to  expect  the  rigors  of  research,  development,  and 
testing  after  Milestone  B  to  uncover  additional  requirements  that  neces¬ 
sitate  additional  funding.  But,  if  we  are  unable  to  completely  avoid  this 
“cost  discovery,”  perhaps  we  should  focus  our  efforts  on  predicting  it.  For 
example,  consider  the  following  questions: 

•  Is  it  true  that  an  Air  Force  fighter  aircraft  program  is  likely 
to  procure  fewer  aircraft  than  originally  planned? 

•  Do  Joint  programs  have  significantly  higher  acquisition 
cost  growth  than  non- Joint  ones? 

•  Is  the  occurrence  of  a  Nunn-McCurdy  breach  in  a  program 
a  good  indicator  of  future  threshold  breaches? 

If  we  are  able  to  hypothesize  a  relationship  between  these  top-level  pro¬ 
gram  characteristics,  then  it  is  possible  to  examine  past  data  to  test  if 
this  relationship  exists.  Furthermore,  if  the  relationship  between  these 
elements  is,  in  fact,  deemed  statistically  and  practically  significant,  then 
we  may  apply  this  relationship  to  correct  estimates  in  new  programs. 
Macro-stochastic  estimation  is  used  to  accomplish  these  goals. 


Macro-Stochastic  Estimation 

To  implement  the  macro-stochastic  estimating  technique  described 
earlier,  we  first  have  to  decide  what  high-level  (macro)  parameters  are 
the  most  strongly  associated  with  cost  estimate  errors.  Next,  we  have 
to  decide  what  constitutes  a  “similar  program”  so  that  we  may  apply  the 
technique  correctly  on  future  data.  In  support  of  these  pursuits,  we  have 
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created  a  database  that  tracks  75  distinct  characteristics  of  MDAPs.2 
The  Selected  Acquisition  Reports  (SAR)  for  these  programs  are  the 
source  for  our  database. 

Programs  that  have  expended  at  least  half  of  their  planned  funding  are 
considered  for  entry  in  the  database  since  these  programs  have  sufficient 
data  to  measure  trends  in  early  program  life.  Also,  only  programs  with 
a  Milestone  B  date  of  1987  or  later  are  included.  This  cutoff  date  allows 
for  a  sufficient  number  of  programs  to  estimate  key  characteristics  and 
also  maintains  some  continuity  and  relevance  with  current  programs 
(Smirnoff  &  Hicks,  2007).  This  filtering  process  results  in  a  sample  of 
937  SARs  describing  70  programs  from  the  Army,  Navy,  and  Air  Force. 
For  each  SAR,  we  compare  the  program's  estimate  of  total  acquisition 
cost  against  the  actual  cost  specified  in  the  program's  final  SAR.  This 
ratio  of  estimated  cost  from  a  particular  SAR  to  the  final  cost  is  defined 
as  the  Cost  Growth  Factor  (CGF).  For  example,  a  program  with  a  CGF  of 
1.3  indicates  that  the  actual  cost  of  the  program  was  30  percent  higher 
than  the  original  estimate.  A  program  that  perfectly  estimated  its  final 
cost  would  have  a  CGF  of  1.0. 

A  statistical  technique  known  as  mixed-model  regression  is  applied  to 
identify  the  parameters  most  strongly  associated  with  changes  in  the 
final  cost  of  a  given  program.  This  advanced  statistical  methodology  is 
required  due  to  the  longitudinal  nature  of  SAR  analysis;  that  is,  repeated 
measurements  of  the  same  program  are  expected  to  be  correlated,  vio¬ 
lating  a  fundamental  assumption  of  basic  linear  regression.  Iteratively 
testing  parameters  in  the  dataset  results  in  an  efficient  model  of  CGF 
containing  the  six  parameters  shown  in  Table  1. 

It  may  seem  like  an  oversight  to  omit  an  explanation  of  how  each  of  these 
parameters  affects  CGF  (that  is,  positively  or  negatively).  In  this  case,  the 
reason  for  this  omission  is  related  to  the  mixed-model  methodology,  and 
would  surely  have  frustrated  former  president  Truman,  as  the  relation¬ 
ship  varies  depending  on  the  program.  Importantly,  these  six  parameters 
are  combined  in  different  ways  to  create  models  tailored  to  specific 
groupings  of  programs,  as  described  in  the  discussion  that  follows. 
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TABLE  1.  SIGNIFICANT  MODEL  PARAMENTERS 

Parameter 

Description 

Fixed/Variable 

Service 

Component 

Identifies  the  executive  military 
service  (Army,  Navy,  or  Air 

Force)  that  leads  the  acquisition 
program.  Marine  Corps  programs 
are  identified  as  belonging  to 
the  Navy. 

Fixed 

Development  to 
Production  Ratio 

The  ratio  of  the  number  of 
years  a  program  spends  in 
development  to  the  number  of 
years  the  program  spends  in 
production. 

Variable 

Count  of 
Development 

APBs 

This  parameter  tracks  the 
number  of  times  a  new  baseline 
is  generated  during  the 
development  phase. 

Variable 

Acquisition  Cost 

The  total  estimated  program 
acquisition  cost,  as  reported 
annually  in  the  SAR. 

Variable 

Quantity  Change 

This  parameter  is  tracked  as  a 
ratio  of  the  procurement  quantity 
planned  in  a  given  year  to  the 
original  Milestone  B  procurement 
quantity. 

Variable 

Year  Count 

The  sequential  numbering  of 
the  program  year,  starting  with 
Milestone  B  as  year  one.  The 
presence  of  this  parameter 
ensures  the  model  is  capable  of 
predicting  the  estimate  trends 
across  time. 

Fixed 

Method 

The  mixed-model  regression  technique  introduces  flexibility  that 
allows  the  analyst  to  generate  different  models  for  different  groupings  of 
programs.  To  return  to  our  hurricane  example,  storms  in  the  Caribbean 
might  behave  differently  than  those  in  the  Atlantic.  This  difference 
may  be  taken  into  account  by  grouping  the  hurricane  data  into  two  bins, 
perhaps  called  Caribbean  and  Atlantic,  and  allowing  the  regression  to 
generate  separate  estimates  according  to  this  partition.  This  feature 
is  very  powerful,  since  it  can  resolve  patterns  that  might  otherwise  be 
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averaged  out  when  the  dataset  is  analyzed  as  a  whole.  More  importantly, 
this  feature  allows  us  to  bin  acquisition  programs  into  groups  according 
to  similarities  in  the  behavior  of  their  cost  estimate  error.  When  we  wish 
to  predict  the  CGF  in  a  new  program,  we  can  apply  the  most  appropriate 
model  of  estimate  errors  by  determining  the  most  suitable  group  for  the 
new  program. 

The  way  programs  are  grouped  is  critical  to  the  predictive  power  of 
the  macro-stochastic  technique.  In  theory,  we  could  put  all  programs 
into  the  same  group;  but  what  we  gain  in  broad  model  applicability,  we 
sacrifice  in  accuracy.  If  the  cost  growth  behavior  for  each  of  these  pro¬ 
grams  was  essentially  the  same,  we  wouldn't  be  so  regularly  thwarted 
when  trying  to  produce  a  useful  budget.  Conversely,  we  could  go  with  the 
opposite  extreme  and  create  a  regression  that  examines  each  program 
individually  by  only  assigning  one  program  to  each  group.  This  grouping 
method  results  in  a  different  model  for  each  program  and  reduces  nearly 
99%  of  the  error  in  program  cost  estimates!  However,  this  accuracy  is 
gained  at  the  expense  of  utility.  Future  programs  cannot  be  assigned 
to  an  existing  group  that  is  uniquely  defined.  The  critical  task,  then,  is 
to  determine  the  most  beneficial  way  to  group  the  programs  in  order  to 
balance  accuracy  with  predictive  capability. 


Program  Grouping 

In  this  study,  programs  are  grouped  according  to  the  categorical 
variables  that  are  most  strongly  correlated  with  the  CGF.  These  variables 
are  simply  characteristics  of  the  program  that  are  known  in  the  first  year, 
and  reported  in  the  first  SAR.  For  example,  final  cost  growth  tends  to  be 
higher  for  new-start  programs  than  programs  that  are  essentially  modifi¬ 
cations  or  variants  of  existing  weapon  systems.  Therefore,  identification 
of  program  iteration  is  used  to  distinguish  program  groupings.  The 
implicit  assumption  with  this  approach  is  that  programs  with  similar 
overall  cost  growth  will  also  exhibit  similar  cost  growth  patterns.  The 
variables  selected  to  bin  programs  are  defined  below. 

1.  Program  Type.  Based  on  the  program  description  in  the 
SAR,  each  program  is  placed  into  one  of  seven  categories: 
Aviation,  Electronic,  Ground  Vehicle,  Maritime,  Munition, 
Space,  and  Space  Launch.  These  categories  are  consistent 
with  previous  program  type  categorizations  (Arena  et  al., 
2006a;  Drezner  et  al.,  1993). 
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2.  Iteration.  This  variable  states  whether  a  program  is  new, 
a  lettered-variant  on  an  existing  program  (e.g.,  the  F-16 
C/D ),  or  a  modification  to  an  existing  program  (e.g.,  the  C-5 
Avionics  Modernization  Program). 

3.  Number  of  Years  Funded.  This  variable  describes  the  num¬ 
ber  of  years  the  program  is  expected  to  be  funded.  This 
variable  may  change  due  to  funding  volatility. 

4.  Joint.  This  binary  variable  indicates  whether  a  program  is 
Joint  between  two  or  more  Services. 

Program  groups  are  created  by  dividing  each  of  the  variables  into  levels, 
ensuring  sufficient  sample  size  within  each  level.  A  program  is  assessed 
a  CGF  “score”  based  on  the  applicable  level  for  each  of  the  four  variables. 
The  program  group  is  the  sum  of  the  CGF  scores  across  the  four  vari¬ 
ables.  Each  program  is  scored  in  this  manner,  and  the  total  scores  from 
each  program  form  the  six  program  groups  shown  in  Figure  4.3 


FIGURE  4.  PROGRAM  GROUPS  RESULTING  FROM  CGF  SCORES 
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Validation  and  Results 

The  mixed-model  regression  uses  the  program  groups  in 
Figure  4  to  fit  different  models  using  the  significant  CGF  pre¬ 
dictors  shown  previously  in  Table  1.  However,  due  to  relatively 
few  programs  in  certain  groups,  validating  the  model  is  neces¬ 
sary  without  omitting  too  many  of  our  samples  for  this 
purpose.  Consequently,  we  validate  the  model  using 
a  technique  that  omits  program  data  in  a  round-robin 
fashion,  predicting  the  CGF  of  the  omitted  program 
and  then  replacing  the  data  to  make  the  prediction  for 
the  next  omitted  program.  This  validation  is  a  type  of  Leave 
One  Out  Cross-Validation  tailored  to  multilevel  or  mixed  mod¬ 
els  (Ryan  et  al.,  2013).  It  results  in  the  aggregation  of  70  separate 
analyses  (one  for  each  program)  into  a  single  set  of  results  that  reflects 
the  expected  predictive  power  of  the  macro-stochastic  model.  The  vali¬ 
dated  model  produces  a  set  of  predicted  CGFs  for  every  program  estimate 
throughout  the  life  of  every  program  in  our  sample.  If  this  version  of  the 
model  is  deemed  reasonably  powerful,  then  the  original  fitted  model  is 
considered  validated  and  is  the  final  model  reported  for  inference. 

Using  the  validated  results,  the  predicted  CGF  for  any  SAR  that  meets 
the  established  completion  criteria  may  be  used  to  correct  the  cost 
estimate  in  that  SAR,  but  some  of  these  corrections  will  be  more  use¬ 
ful  than  others.  Since  the  SAR  estimates  get  progressively  better  over 
time,  there  is  equivalently  less  CGF  error  for  the  model  to  correct,  thus 
reducing  the  average  predictive  performance  of  the  model  as  a  program 
matures.  Consequently,  the  macro-stochastic  technique  is  most  useful 
when  applied  to  correct  the  earliest  cost  estimates  in  a  program.  In  fact, 
for  each  additional  percentage  of  program  expenditure,  the  model  loses 
approximately  three-quarters  of  a  percent  of  its  predictive  power. 

The  70  programs  in  our  dataset  displayed  a  mean  CGF  of  1.44,  measured 
from  the  initial  SAR  estimate.  This  means  that  the  programs  underesti¬ 
mated  their  eventual  cost  by  44%,  on  average.  However,  this  is  an  average 
of  underestimates  and  overestimates.  For  the  purposes  of  resource  allo¬ 
cation,  under  and  overestimation  of  budgetary  requirements  may  both 
be  considered  detrimental  because  dollars  allocated  to  one  program  can¬ 
not  be  easily  transferred  to  another.  Since  the  model  seeks  to  minimize 
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cost  estimate  errors  regardless  of  direction,  the  absolute 
estimate  error  is  a  more  appropriate  measure.  Our  sample 
showed  a  mean  absolute  error  of  57%. 

In  contrast,  after  applying  the  macro-stochastic 
technique,  the  model-corrected  CGF  for  these 
initial  estimates  averaged  0.93— slightly  over¬ 
estimating,  but  closer  to  the  ideal  1.0  CGF.  As 
shown  in  Figure  5,  the  average  absolute  error  for  model- 
corrected  estimates  was  27%,  representing  a  19%  reduction  in 
the  average  absolute  cost  estimate  error,  across  all  programs.  However, 
model  performance  is  best  in  early  program  life;  the  average  error  reduc¬ 
tion  in  the  first  estimate  is  37%.  Also,  since  the  six  program  groups  are 
assigned  by  assessing  the  severity  of  their  cost  growth,  we  expect  that 
the  most  significant  improvement  will  be  seen  when  the  model  is  applied 
to  the  “high-growth”  programs.  When  the  algorithm  is  applied  to  the  first 
estimate  of  programs  in  CGF  categories  four  through  six,  90%  of  these 
estimates  are  improved,  with  an  average  error  reduction  of  45%. 


FIGURE  5.  SUMMARY  OF  VALIDATED  MODEL  PERFORMANCE 
ACROSS  ALL  70  PROGRAMS 
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Reporting  model  performance  as  a  percent  improvement  is  useful 
because  it  normalizes  programs  of  disparate  cost.  However,  since  our 
research  focuses  on  real  dollars,  it  is  important  to  convert  the  percent 
error  reduction  into  a  dollar  amount  to  demonstrate  model  efficacy.  The 
absolute  percent  error  for  each  program  is  multiplied  by  its  final  cost  and 
converted  to  base  year  2013  dollars  in  order  to  establish  the  total  dollar 
amount  reallocated  by  the  validated  model.  The  aforementioned  19% 
reduction  in  error  equates  to  $119.5  billion,  in  base  year  2013  dollars. 
If  the  total  cost  of  these  programs  is  scaled  to  equal  that  of  the  current 
DoD  MDAP  portfolio  (DoD,  2013),  then  this  macro-stochastic  model 
could  potentially  allocate  $6.24  billion  more  efficiently  every  year,  if 
consistently  applied  to  the  first  estimate  of  new  MDAPs. 


What  This  Technique  Is  Not 

These  results  clearly  illustrate  the  utility  of  the  macro-stochastic 
cost-estimating  approach.  But,  as  is  often  the  case  with  statistical  tools, 
it  is  perhaps  equally  important  to  manage  expectations  by  explaining  a 
few  of  the  applications  for  which  this  technique  is  ill-suited. 

1.  Adjusting  cost  estimates  at  the  program  office  level.  The 
efficacy  of  the  model  deteriorates  rapidly  and,  even  when 
applied  to  the  first  estimate  of  every  program,  only  about 
72%  of  program  estimates  are  improved.  This  notion  that 
estimates  are  only  improved  on  average  can  be  a  significant 
source  of  doubt  when  it  suggests  that  a  program's  rigorously 
developed  estimate  might  be  44%  too  low.  However,  the 
average  cost  of  programs  is  sufficient  for  informing  bet¬ 
ter  affordability  decisions  when  considering  a  portfolio  of 
assets. 

2.  Placing  blame  and  driving  acquisition  reform.  Macro¬ 
stochastic  estimation  eschews  the  typical  cause-and-effect 
relationship  that  so  many  other  acquisition  studies  seek  to 
uncover.  Rather,  the  model  draws  its  power  from  the  cor¬ 
relation  between  seemingly  unrelated  things.  For  example, 
it  would  be  incorrect  to  say  that  the  Service  Component 
causes  cost  growth;  it  is  simply  an  observed  correlation. 

This  lack  of  causality  makes  this  model  ill-suited  for  sug¬ 
gesting  changes  to  the  acquisition  process. 
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3.  Placing  bounds  on  a  traditional  cost  estimate.  The  full  text 
of  this  study  (DeNeve,  2014)  explains  the  prediction  inter¬ 
vals  that  surround  the  estimates  of  CGF.  However,  these 
alone  do  not  constitute  the  “cone  of  uncertainty”  discussed 
earlier  in  this  article.  With  changes  to  the  APB,  the  distri¬ 
bution  around  the  predicted  CGF  and  the  cost  estimate 
will  change.  Both  of  these  distributions  must  be  taken  into 
account  when  placing  bounds  on  the  model-corrected  final 
cost  estimate.  This  is  a  subject  for  future  work. 


Conclusions 


The  existing  paradigm  for  reporting  acquisition  cost  based  on  a  fixed 
APB  results  in  unrealistic  budgets  and  chronically  inefficient  resource 
allocation.  In  the  current  environment  of  fiscal  restraint,  embracing 
uncertainty  can  help  provide  a  more  realistic  view  of  a  program's  true 
affordability.  Acknowledging  the  likelihood  of  changes  to  a  program's 
baseline  grants  the  freedom  to  leverage  past  data  and  predict  trends 
in  cost-estimate  performance.  While  not  suitable  as  a  low-level  cost 
estimating  tool,  this  study  demonstrates  such  a  method  to  reduce  cost- 
estimate  error  in  the  earliest  estimates  of  major  defense  programs, 
helping  to  stabilize  long-term,  portfolio-level  budgets.  As  demonstrated 
by  Figure  5,  our  model  achieves  the 
most  significant  error  reduction  early  in 
program  life,  when  accurate  estimates 
are  crucial  for  resource  allocation  and 
affordability  decisions.  In  fact,  nearly 
half  of  the  estimate  error  is  reduced 
when  the  model  is  applied  early  to  the 
most  growth-prone  acquisition  pro¬ 
grams.  As  with  hurricane  forecasting, 
the  optimal  approach  for  acquisition 
cost  estimation  is  likely  a  combination 
of  techniques  that  focuses  on  providing 
the  most  useful  estimate,  even  if  this 
means  embracing  the  uncertain  nature 
of  defense  acquisition. 
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Endnotes 

The  Defense  Acquisition  Guidebook  dictates  that  MDAPs  “must  state  the  confidence 
level  used  in  establishing  a  cost  estimate... in  the  next  Selected  Acquisition  Report 
prepared  in  accordance  with  10  U.S.C.  §  2423”  (DAU,  n.d.,  Chap  3,  §3.4.1).  The 
referenced  section  of  U.S.  Code  contains  no  such  requirement,  and  few  SARs  currently 
report  confidence  in  their  estimates. 

2MDAPs  are  the  largest  programs  in  the  DoD,  defined  by  having  more  than  $509 
million  for  Research,  Development,  Test  &  Evaluation,  or  more  than  $3  billion  for 
procurement  in  Base  Year  2010  dollars  (Weapon  Systems  Acquisition  Reform  Act, 
2009).  In  fiscal  year  2014,  MDAPs  constituted  40  percent  of  the  acquisition  funding 
for  the  DoD  (DoD,  2013)  and  since  1969,  they  have  been  required  to  submit  a 
standardized  annual  report  of  their  status,  called  the  Selected  Acquisition  Report 
(GAO,  2012b). 

This  scoring  methodology  is  explained  in  far  greater  detail  in  the  full  text  of  the  study 
(DeNeve,  2014). 
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The  inability  of  Department  of  Defense  (DoD)  programs  to  sufficiently  reduce 
technology  risk  prior  to  entering  formal  systems  development  has  between 
2007  and  2012  contributed  to  a  13  percent  cost  growth  in  weapon  systems 
acquisition  and  a  17  percent  increase  in  cycle  time  to  deliver  initial  operational 
capability  With  the  advent  of  key  legislation  and  resulting  DoD  acquisition 
reform  initiatives,  weapon  systems  programs  are  now  required  to  enforce  a 
technology  development  strategy  that  can  foster  true  risk  reduction  prior  to 
entering  systems  development.  A  key  enabler  to  reducing  technology  risk  and 
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thereby  accelerating  design  maturity  is  the  use  of  system  prototype  demon¬ 
strations.  The  objective  of  this  article  is  to  present  research  findings  on  the 
“effects  of  system  prototype  demonstrations  on  weapons  systems  develop¬ 
ment”  for  major  defense  acquisition  programs.  The  results  of  this  research 
will  better  inform  systems  engineers  and  contribute  to  improved  technology 
development  strategy. 


Keywords:  system  prototype,  technology  demonstration,  technology  maturity,  design  maturity, 
program  performance 
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The  Department  of  Defense  (DoD)  has  historically  struggled  to 
implement  effective  risk-mitigation  strategies  in  the  development  of 
highly  complex  weapon  systems,  as  evidenced  by  increasing  cost  and 
schedule  growth  over  the  past  several  decades  (General  Accounting 
Office,  1999;  Government  Accountability  Office  [GAO],  2006b).  The 
inability  of  DoD  programs  to  sufficiently  reduce  technology  risk  prior 
to  allowing  a  program  to  enter  formal  systems  development  has,  as 
measured  from  2007  to  2012,  contributed  to  a  13%  cost  growth  in 
weapon  systems  acquisition,  and  a  17%  increase  in  cycle  time  to  Initial 
Operational  Capability,  or  IOC  (GAO,  2013).  Acquisition  cycle  time  is 
defined  as  that  span  of  time  from  program  start  to  deployment  of  IOC  to 
the  warfighter.  When  compared  to  First  Full  Estimates,  the  DoD  major 
defense  acquisition  program  (MDAP)  portfolio  total  acquisition  cost  had 
grown  an  average  38%;  correspondingly,  product  cycle  time  increased  an 
average  37%  (GAO,  2013). 

First  Full  Estimates,  as  defined  by  the  GAO,  are  the  original  total  acqui¬ 
sition  cost  estimates  established  at  program  development  start  (GAO, 
2012,  p.  36).  The  GAO  estimates  for  MDAPs  and  their  total  acquisition 
costs  are  collected  from  DoD  Selected  Acquisition  Reports  (SAR)  and 
consist  of  research  and  development,  operations  and  maintenance,  and 
military  construction  costs  (GAO,  2012,  p.  171).  Clearly,  this  performance 
trend  has  been  unacceptable,  and  further  attention  is  required  to  manage 
technology  risk  effectively. 

Today's  economic  climate  continues  to  threaten  available  DoD  funds 
and  underscores  the  need  for  streamlined  but  effective  systems  engi¬ 
neering.  Smart  application  of  cost-effective  tools  and  techniques,  such 
as  the  use  of  system  prototype  demonstrations,  should  be  leveraged  to 
ensure  maximum  payback  per  dollar  towards  risk  reduction.  The  cost  of 
using  prototypes,  balanced  with  value-added  risk-reduction  returns,  will 
contribute  to  program  “Should  Cost"  savings.  The  phrase  Should  Cost, 
institutionalized  by  DoD  as  part  of  Better  Buying  Power  2.0,  is  an  initia¬ 
tive  for  MDAPs  to  eliminate  inefficiencies  and  capitalize  on  cost-saving 
opportunities  (Carter  &  Mueller,  2011).  In  a  recent,  concise,  and  highly 
convincing  article  published  in  Proceedings,  the  U.S.  Naval  Institute's 
flagship  magazine,  VADM  David  Dunaway,  Commander  of  the  Naval  Air 
Systems  Command,  wrote  about  today's  economic  climate:  “In  the  face 
of  decreasing  budgets,  rapidly  evolving  threats,  and  a  shift  in  defense 
strategy, ...  it's  imperative  that  every  dollar  spent  increases  warfighting 
capability"  (Dunaway,  2013,  p.  326). 
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Today's  economic  climate  continues  to  threaten 
available  DoD  funds  and  underscores  the  need  for 
streamlined,  but  effective  systems  engineering . 


Through  the  use  of  descriptive  statistics  and  empirical  analysis,  this 
article  summarizes  the  comparative  performance  for  MDAPs  that  did 
and  did  not  invest  in  system  prototype  demonstrations  for  early  risk 
reduction  prior  to  entering  system  development,  otherwise  referred  to 
as  Engineering  and  Manufacturing  Development  (EMD).  Additionally, 
for  those  MDAPs  that  did  use  prototype  demonstrations  over  this  past 
decade,  program  performance  was  examined  for  any  impacts  coin¬ 
cident  with  the  adoption  of  related  key  systems  engineering  policy 
and  legislation. 

With  the  Defense  Acquisition  Management  System  (DAMS)  model  as 
a  conceptual  framework,  key  hypotheses  were  evaluated  using  empiri¬ 
cal  analysis  of  historical  evidence  and  trends  to  help  validate  observed 
system  behavior.  The  effects  of  pre-EMD  system  prototype  demonstra¬ 
tions  on  program  performance  were  examined  using  observed  impacts 
to  technology  readiness  and  weapon  system  design  maturity.  The  data 
analysis  does  not  highlight  any  individual  program  specifics,  but  applies 
a  macro-level  analysis  of  aggregated  data  to  characterize  observed  pro¬ 
gram  performance  as  a  function  of  key  predictor  variables. 

The  authors  anticipate  that  the  findings  of  this  research  would  help  to  (a) 
better  inform  program  managers  and  systems  engineers  on  the  effects  of 
system  prototype  demonstrations  on  weapon  systems  development;  (b) 
better  provide  insightful  knowledge  to  develop  more  effective  technol¬ 
ogy  development  strategy;  and  (c)  better  implement  “true”  risk  reduction 
measures,  per  DoD  guidance  (Kendall,  2012)  before  entering  the  EMD 
phase.  The  context  of  “true”  in  reference  to  risk  reduction  is  meant  to 
imply  pre-EMD  system  development  mitigation  activities  that  can  indeed 
reduce  the  risk  of  cost  and  schedule  growth,  and  minimize  product 
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cycle  time  to  the  warfighter.  System  prototype  demonstrations  not  only 
validate  the  state  of  technology  maturity  for  enabling  technologies,  but 
also  provide  for  early  mitigation  of  system/subsystem  integration  risk. 
Consonant  with  DoD's  goals  for  improving  Better  Buying  Power,  this 
research  also  provides  additional  insight  into  whether  perceived  gains 
from  pre-EMD  prototype  demonstrations  are  actually  being  realized. 


Prototype  Demonstrations 
—A  Historical  Perspective 

As  demonstrated  in  the  early  1900s,  whether  it's  the  Wright  brothers' 
experimentation  leading  up  to  the  first  successful  flight  of  the  Wright 
Flyer,  or  Samuel  Langley's  attempts  to  launch  an  Aerodrome  for  the  first 
time  off  a  modified  houseboat  at  sea,  our  nation's  industry  has  leveraged 
system  prototype  demonstrations  for  over  a  century.  Figure  1  portrays 
two  historical  moments  in  time  where  system  prototypes  were  used  to 
reduce  early  aviation  technology  risk. 

Prototypes  provide  the  designer  a  useful  tool  with  which  to  visualize 
and  transition  new  ideas  into  development  using  an  archetype,  initial 
model,  or  early  pattern  of  the  envisioned  end  product.  Industry  has  lev¬ 
eraged  prototypes  with  great  success  as  a  necessary  enabler  and  bridge 
to  introduce  new  products  into  the  marketplace.  Although  the  value  of 
prototypes  may  seem  obvious,  historically  the  use  of  prototypes  and 
the  perceived  return  on  investment  has  been  a  subject  of  debate.  The 
following  chronology  highlights  DoD's  changing  opinion  on  the  use  of 
prototype  demonstrations: 

•  (Favorable)  As  early  as  the  1930s,  industry  commonly  built 
engine-aircraft  combination  prototypes  as  a  form  of  air¬ 
craft  development  risk  mitigation  (Drezner,  1992). 

•  (Favorable)  Post-World  War  II,  in  the  mid  to  late  1940s, 
competitive  prototype  flight  testing  occurred  with  the 
transition  of  propellers  to  reciprocating  engines  (Smith, 
Barbour,  McNaugher,  Rich,  &  Stanley,  1981). 

•  (Not  Preferred)  In  the  1950s,  since  prototypes  were  not 
representative  of  full-scale  development  integrated  designs, 
the  opinion  was  that  the  practice  was  wasteful  and  non¬ 
value  added  (Smith  et  al.,  1981). 
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FIGURE  1.  EARLY  EXAMPLES  OF  AVIATION  SYSTEM 
PROTOTYPE  DEMONSTRATIONS 


Samuel  Langley's  Aerodrome,  December  8,  1903 


Wilbur  Wright's  1st  Successful  Flight,  December  17,  1903 


Note.  Photos  courtesy  Library  of  Congress  (Smithsonian  Libraries,  n.  d.  a  and  b). 
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•  (Not  Preferred)  With  the  advent  of  the  digital  computer  age 
in  the  1960s,  a  prevailing  philosophy  existed  that  theoreti¬ 
cal  analysis  would  be  sufficient  to  predict  systems  design 
performance  without  the  need  for  costly  prototypes  (Smith 
et  al.,  1981). 

•  (Favorable)  Coincident  with  the  first  issuance  of  DoD 
Directive  5000.1  in  1971,  prototyping  was  re-introduced 
as  a  key  risk  reduction  tool  as  a  result  of  then-Secretary  of 
Defense  David  Packard's  “Fly-Before-Buy”  promulgated 
policy.  Competitive  prototypes  were  encouraged  with  less 
dependence  on  concurrent  development  and  paper  studies 
before  entering  Full-Scale  Development  (DoD,  1986). 

•  (Favorable)  In  1986,  the  President's  Blue  Ribbon 
Commission  on  Defense  Management,  referred  to  as  the 
Packard  Commission,  reported  the  need  for  rigorous  testing 
of  system  prototypes  prior  to  Full-Scale  Development,  again 
emphasizing  a  Fly-Before-Buy  philosophy  (DoD,  1986). 
Subsequent  legislation  was  introduced  in  1987,  which  man¬ 
dated  that  DoD  develop  and  test  competitive  prototypes 
for  MDAPs  before  awarding  a  production  contract  (Glass, 
1988). 

•  (Favorable)  As  a  result  of  a  General  Accounting  Office 
(1999)  study  rcommendation,  in  2001  DoD  adopted  the 
use  of  Technology  Readiness  Levels  (TRL)  as  a  means  for 
MDAPs  to  manage  the  maturity  of  technology  entering 
system  development  (Technology  Readiness,  2010).  The 
National  Defense  Authorization  Act  (NDAA)  of 2006  estab¬ 
lished  statutory  law  for  the  Milestone  Decision  Authority 
to  certify  that  all  critical  technologies  (i.e.,  referred  to  as 
critical  technology  elements)  have  been  demonstrated  in 
a  relevant  environment  (i.e.,  TRL  6)  before  granting  an 
MDAP  approval  to  enter  EMD  (NDAA,  2006).  In  2007,  then- 
Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics  John  Young  released  a  memorandum, 
“Prototyping  and  Competition,”  directing  the  Services 
and  Defense  Agency  proponents  for  MDAPs  to  “formulate 
all  pending  and  future  programs  with  acquisition  strate¬ 
gies  and  funding  that  provide  for  two  or  more  competing 
teams  producing  prototypes  through  milestone  (MS)  B” 
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(Young,  2007).  The  Weapon  Systems  Acquisition  Reform 
Act  of 2009  (WSARA)  introduced  legislation  that  enforced 
specific  risk-reduction  efforts  prior  to  entering  system 
development,  including  engagement  with  industry  before 
EMD  for  technology  maturation;  competitive  prototyping; 
and  the  establishment  of  a  system  allocated  baseline  at  a 
system-level  Preliminary  Design  Review  (WSARA,  2009). 

What  Constitutes  a  Prototype? 

The  term  "prototype"  has  many  definitions  depending  on  the  context 
and  need.  First,  it  is  important  to  understand  the  difference  between 
prototyping  and  a  prototype.  In  general,  prototyping  is  a  process  to  fos¬ 
ter  creativity  and  new  ideas,  visualize  novel  application  and  enabling 
technologies,  reduce  uncertainty  and  increase  the  advancement  of 
knowledge,  and  highlight  the  art  of  the  possible.  Prototypes  provide  the 
mechanism  to  “uncover  truth”  (National  Research  Council,  2013,  p.  3) 
through  observed  and  controlled  experiments  that  allow  for  the  col¬ 
lection  of  quantifiable  data  to  explore,  develop,  validate,  and  improve 
performance  prediction  models  or  theories. 

The  primary  purpose  for  using  a 
prototype  is  to  mitigate  risk  (cost, 
schedule,  or  performance)  to  prod¬ 
uct  development  and  to  the  timely 
delivery  of  an  affordable  and  com¬ 
pliant  end-item  to  the  customer. 

Prototypes  focus  on  high-risk 
areas  considered  essential  to 
achieve  system  performance  and 
are  deemed  important  to  achieve 
market  or  user  introduction.  The 
cost  and  relative  complexity  that 
a  prototype  can  take  on  will  vary 
depending  on  the  need  and  the 
significance  of  the  function  being 
mitigated.  From  small-scale,  rela¬ 
tively  simple  models  for  desktop 
experiments  to  larger,  more  complex  full-scale  integrated  system  demon¬ 
strators,  the  primary  goal  for  the  use  of  a  prototype  is  to  yield  insightful 
knowledge  that  can  be  used  to  reduce  end-item  risk. 
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A  prototype  fundamentally  is  used  to  demonstrate  increasing  levels  of 
system  integrated  solutions  in  stages  of  representative  environments 
to  meet  expected  operational  performance  in  mission-relevant  sce¬ 
narios.  When  considering  the  general  nature  of  prototyping,  a  RAND 
Corporation  study  (Drezner,  1992)  concluded  that  a  prototype  is  best 
defined  as: 

...  a  product  (hardware  and/or  software)  that  allows 
hands-on  testing  in  a  realistic  environment.  In  scope  and 
scale,  it  represents  a  concept,  subsystem,  or  production 
article  with  potential  utility.  It  is  built  to  improve  the 
quality  of  decisions,  not  merely  to  demonstrate  satisfac¬ 
tion  of  contract  specifications,  (p.  9) 

Criticality  of  Prototype  Demonstrations 
on  Technology  Maturity 

The  term  “maturity”  or  “technology  maturity”  refers  to  that  period 
in  which  an  enabling  technology  translates  from  instantiation  of  an  idea 
to  the  realization  of  that  idea's  fullest  potential.  The  product  life  cycle 
therefore  transitions  from  early  conceptual  and  technology  development, 
through  systems  development  (i.e.,  Developmental  Test  and  Evaluation), 
operational  test,  production,  market  or  user  introduction,  and  finally,  to 
disposal  or  recycle. 

Maturity  is  a  relative  term  that  is  applied  based  on  comparison  to  a 
predefined  end  state.  When  discussing  the  readiness  to  enter  system 
development,  a  technology  that  has  not  achieved  TRL  6  is  considered 
“immature.”  According  to  DoD  (DoD,  n.d.;  Taylor,  2007)  and  Public  Law 
(NDAA,  2006,  2008),  technologies  that  are  TRL  6  or  better  are  consid¬ 
ered  as  meeting  the  minimum  maturity  level  acceptable  to  enter  system 
development  (i.e.,  EMD)  at  Milestone  B.  When  considering  a  production 
decision  at  Milestone  C,  DoD  best  practice  requires  technologies  to  be 
at  least  TRL  7  to  be  considered  mature  enough  to  enter  a  production 
decision.  A  similar  relationship  applies  when  considering  readiness 
for  deployment;  those  technologies  not  yet  TRL  8  (i.e.,  fully  qualified, 
specification-compliant,  and  ready  to  enter  operational  test)  would  not  be 
considered  mature  enough  to  enter  the  capstone  Operational  Evaluation 
(OPEVAL).  Although  GAO  and  DoD  agree  that  any  critical  technol¬ 
ogy  less  than  TRL  6  is  considered  “immature,”  GAO  recommends  that 
TRL  7,  not  TRL  6,  is  the  appropriate  level  of  technology  maturity  when 
entering  product  development  (i.e.,  EMD  or  GAO  Knowledge  Point  #2). 
GAO  refers  to  critical  technologies  at  TRL  6  as  “approaching  or  nearing 
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maturity”  DoD  considers  TRL  9  as  the  level  when  a  critical  technology 
can  be  considered  fully  mature  (i.e.,  when  the  system  is  considered  suit¬ 
able  and  effective  by  the  user  and  deployed  to  field).  GAO,  on  the  other 
hand,  considers  critical  technologies  as  “mature  or  fully  mature”  at 
TRL  7  when  a  production  decision  at  Milestone  C  is  required  (i.e.,  GAO 
Knowledge  Point  #3;  GAO,  2006a,  p.  132). 

Figure  2  associates  the  level  of  prototype  and  demonstrations,  the  venue 
for  those  demonstrations,  and  the  technology  maturity  achieved  as 
delineated  by  assigned  TRLs  to  the  applicable  dimension  of  the  DoD 
acquisition  life  cycle.  The  diagram  shows  that  as  Science  and  Technology 
(S&T)  progresses  from  early  exploratory  development  (i.e.,  basic  prin¬ 
ciples,  analytical  studies,  and  early  experimentation)  to  the  formulation 
and  test  of  component/breadboard  prototypes  in  a  low-fidelity  laboratory 
environment,  the  product  performance  (i.e.,  demonstrated  technology 
maturity)  curve  exhibits  a gradual-to-exponential  growth  (TRL  1  to  TRL 
4).  After  entering  Milestone  A  (i.e.,  Technology  Maturation  and  Risk 


FIGURE  2.  LEVEL  OF  PROTOTYPE  DEMONSTRATIONS,  VENUE, 
AND  TECHNOLOGY  MATURITY 


q> 


Materiel  Solution  Analysis  TDorTMRR  EMD  Production  &  Deployment 

(Early  S&T)  (S&T  Transition  Opportunity)  (System  Development) 


Note.  EMD  =  Engineering  and  Manufacturing  Development;  TD  =  Technology  Develop¬ 
ment;  TMRR  =  Technology  Maturation  and  Risk  Reduction;  TRL  =  Technology  Readi¬ 
ness  Level;  S&T  =  Science  and  Technology. 
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Reduction  phase),  the  curvature  becomes  less  steep  over  an  extended 
period  of  technology  development  as  competitive  prototype  solutions  are 
used  to  demonstrate  critical  technologies  in  a  relevant  environment  (i.e., 
TRL  6).  Upon  achieving  a  TRL  6  level  of  maturity,  a  more  gradual  inclin¬ 
ing  plateau  results  for  the  duration  of  EMD.  This  flatter  profile  indicates 
a  lower  technological  risk  exists  (i.e.,  related  to  technology  maturity)  and 
a  representative  system  prototype  or  model  of  the  end-state  product  has 
been  achieved.  During  EMD,  there  should  be  no  more  reliance  on  S&T; 
only  standard  engineering  developmental  test  and  evaluation  should 
be  applied  both  to  finish  product  design  and  build/test  a  production 
representative  prototype  (i.e.,  engineering  development  model)  prior  to 
Milestone  C.  After  the  actual  system  has  been  fielded  and  the  technol¬ 
ogy  eventually  approaches  end-of-life,  the  tail  of  the  flattened  S-curve 
dips,  reflecting  technology  aging  as  well  as  a  degradation  in  both  system 
reliability  and  supportability. 

As  shown  in  Figure  2,  the  S-curve  shape  represents  a  generic  depiction 
of  increasing  technology  maturity  and  product  performance  over  time 
while  progressing  through  the  acquisition  life  cycle.  Several  analogies 
have  been  theorized  relating  technology  maturity  with  the  shape  and 
phenomenon  of  an  S-curve  (MITRE,  n.  d.;  Nolte,  2008).  Although  the 
shape  of  the  curve  implies  a  changing  rate  of  improving  maturity  or 
product  utility  consistent  with  increasing  levels  of  integrated  prototype 
demonstrations  and  development  progress,  the  overlaid  TRL  mapping 
shown  in  the  figure  should  be  interpreted  as  discrete  threshold  attain¬ 
ment  points  where  increasing  levels  of  technology  maturity  can  be 
claimed.  TRL  values  are  assigned  only  as  integer  values  (i.e.,  DoD  does 
not  recognize  a  readiness  level  fraction).  Only  when  enough  aggregate 
demonstration  evidence  of  technology  maturation  has  been  collected 
can  the  Technology  Readiness  Assessment  (TRA)  independent  review 
panel  substantiate  assignment  of  the  next  integer  TRL  value.  The  TRL 
definitions,  demonstration  criteria,  and  TRL  values,  as  overlaid  onto  the 
S-curve  and  shown  in  Figure  2,  are  consistent  with  DoD  guidance  and 
policy  (DoD,  n.d;  DoD,  2011). 

Key  Aspects  of  Prototype  Demonstrations 

The  applicable  venues  for  the  demonstration  of  a  prototype  depend  on 
the  level  of  information  required,  complexity  and  integration  level  of  the 
prototype,  relevant  environment  in  which  the  prototype  must  operate, 
performance  expectations,  and  the  technology  maturity  required  at  the 
associated  stage  within  the  DoD  acquisition  life  cycle.  Considerations 
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of  potential  relevant  environments  for  which  a  critical  technology 
would  need  to  survive  and  meet  operational  performance  would  include 
physical,  logical,  data,  security,  and  user.  The  relevant  environment  is 
characterized  by  the  critical  technology  application  and  its  operational 
performance  expectations  while  under  worst-case,  mission-relatable 
conditions. 

A  Critical  Technology  Element  (CTE)  represents  an  enabling  technology 
that  is  deemed  critical  to  meet  operational  performance  of  the  system 
to  be  acquired  and  is  also  (a)  a  technology  or  application  of  a  technology 
that  is  considered  either  new  or  novel,  or  (b)  represents  an  area  that  poses 


FIGURE  3.  TECHNOLOGY  READINESS  LEVEL  MAPPING  TO 
PROTOTYPE  DEMONSTRATION  ATTRIBUTES 


Compliant,  Qualified,  &  Test/Demo  Complete 

(Ready  for  Operational  Evaluation,  OPEVAL) 

Demonstrated  in  Operational  Environment 

(Ready  for  Limited  Production  Decision,  MSC) 


Demonstrated  in  Relevant  Environment 

(Ready  to  Enter  System  Development,  MSB) 


Validated  in  Relevant  Environment 

(Ready  to  Enter  Technology  Development,  TMRR) 

Validation  in  Laboratory  Environment 

(Ready  to  Begin  Bridge  for  Technology  Transition) 

Analytical  Validation  &  Proof  of  Concept 

(Start  Active  Research  &  Development,  R&D) 

Early  Studies  for  Application  Formulation 

(Invention  &  Practical  Application  Begins) 

Technology  Research 

(Pure  Science  Begins  Translation  to  R&D) 


(DoD,  2009;  DoD,  2011) 


Note.  EMD  =  Engineering  and  Manufacturing  Development;  MSC  =  Milestone  C; 
MSB  =  Milestone  B;  OPEVAL  =  Operational  Evaluation;  P&D  =  Production  &  De¬ 
ployment;  R&D  =  Research  &  Development;  TMRR  =  Technology  Maturation  &  Risk 
Reduction. 


a  significant  technological  risk  during  product  development  (i.e.,  EMD) 
(DoD,  n.d.;  DoD,  2009).  A  TRA  is  conducted  using  an  independent  review 
panel  to  reconcile  program  CTEs  and  associate  TRLs  based  on  the  level 
and  quality  of  integrated  prototype  demonstrations  accomplished. 
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Figure  3  provides  a  mapping  of  TRL  descriptions  and  definitions  to  pro¬ 
totype  demonstration  environment  and  venue,  level  of  technology,  and 
expected  attainment  across  the  DAMS  timeline. 


Conceptual  Framework 

For  this  study,  a  research  conceptual  framework  was  established  to 
examine  the  effects  that  system  prototype  demonstrations,  when  applied 
early  in  the  systems  engineering  acquisition  life  cycle,  would  have  on 
reducing  technology  risk  for  system  development  and  production  of  U.S. 
military  weapon  systems.  Since  the  approach  leverages  event-driven 
knowledge  points  (e.g.,  design  reviews)  consistent  with  standard  sys¬ 
tems  engineering  practice,  the  framework,  as  applied,  can  be  tailored  to 
accommodate  other  agency  or  industry  product  life  cycles.  The  DAMS 
is  a  disciplined  systems  engineering,  event-based  framework  in  which 
acquisition  programs  proceed  through  a  series  of  milestone  decision 
reviews  for  authorization  to  enter  subsequent  life-cycle  phases  of  the 
weapon  systems  acquisition  process  (DoD,  2013).  Relationships  were 
examined  between  key  variables  related  to  technology  maturity,  design 
maturity,  and  their  corresponding  impact  on  program  performance. 

The  DAMS  provided  the  rigorous  structure  necessary  to  collect  and 
analyze  descriptive  statistics  on  independent  variable  constituents 
representing  technology  and  design  maturity,  as  well  as  on  program 
performance  dependent  variables  (i.e.,  cost,  schedule,  and  product  cycle 
time).  Today's  prevailing  best  practices  endorse  the  use  of  system  proto¬ 
type  demonstrations  as  a  major  contributor  to  true  risk  reduction  before 
entering  system  development  (Carter,  2010;  Kendall,  2012;  Young,  2007). 
In  fact,  DoD's  expectations/assumptions  now  encompass  realization  of 
not  only  reduced  program  cost  and  schedule  growth,  but  shorter  prod¬ 
uct  cycle  time  to  the  warfighter.  The  following  questions  were  used  to 
examine  the  validity  of  these  assumptions: 

•  Do  technology  development  (i.e.,  pre-EMD)  system  pro¬ 
totype  demonstrations  provide  a  positive  return  on 
investment  for  weapon  systems  development? 

•  Do  technology  development  system  prototype  demonstra¬ 
tions  impacting  technology  maturity  improve  weapon 
systems  development  program  performance? 
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•  Do  technology  development  system  prototype  demonstra¬ 
tions  have  a  positive  impact  on  achieving  weapon  systems 
design  maturity? 

Research  Population  and  Sampling  Description 

The  research  population,  consisting  of  DoD  MDAP  portfolios 
ranging  from  FY  2002  through  FY  2012,  were  designated  Acquisition 
Category  I  (ACAT-I)  since  they  were  projected  to  exceed  threshold  FY 
2000  constant  dollars  criteria  for  either  Research,  Development,  Test 
and  Evaluation  ($365  million)  or  Procurement  ($2.19  billion)  (DoD, 
2000, 2008).  The  latest  interim  DoDI  5000.02  (DoD,  2013)  modified  the 
ACAT-I  designation  criteria  to  be  relative  to  FY  2014  constant  dollars  for 
subsequently  established  MDAPs.  A  mixed-methods  research  approach 
was  used  to  collect  and  analyze  historical  program  performance  data 
and  findings  from  available  and  relevant  literary  sources.  Data  collection 
was  focused  primarily  on  MDAPs  that  were  part  of  the  annually  pub¬ 
lished  GAO  assessments  for  selected  major  weapon  systems  programs. 
These  reports,  dating  from  2003  to  2013,  represent  limited  case  study, 
knowledge-based  program  performance  assessments  that  were  provided 
to  the  United  States  Congress.  The  actual  data  contained  within  these 
published  reports  are  mostly  reflective  of  the  previous  year's  program 
performance,  therefore  representing  MDAP  portfolios  spanning  from 
2002  to  2012.  MDAP  cost,  schedule,  and  performance  data  were  also  col¬ 
lected  from  annual  DoD  SARs,  which  are  submitted  in  conjunction  with 
the  President's  Budget.  The  research  data  population  consisted  solely  of 
MDAPs  and  did  not  include  Major  Automated  Information  Systems,  or 
ACAT-I  A  programs. 


Considerations  of  potential  relevant  environments 
for  which  a  critical  technology  would  need  to 
survive  and  meet  operational  performance  would 
include  physical,  logical,  data,  security,  and  user. 
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After  initial  data  cleansing  to  ensure  validity  and  reliability,  139  MDAPs 
were  determined  to  contain  enough  usable  and  relevant  data  for  analy¬ 
sis  of  key  research  factors  of  interest.  Considerations  used  for  data 
purification  included  adequacy  of  sample  size,  verification  of  ACAT 
assignment,  and  noting  if  programs  were  canceled  or  restruc¬ 
tured.  The  research  population  spread  was  as  follows:  25% 

Air  Force  (34  MDAPs),  23%  Army  (32  MDAPs),  35%  Navy 
and  Marines  (49  MDAPs),  and  17%  DoD  Joint  (24  MDAPs). 

Product  types  included  aircraft,  helicopters,  satellites, 
ships,  submarines,  ship/ground  vehicles,  ship/ground 
stations,  sensors  and  electronic  warfare  systems, 
missiles,  weapons  and  munitions,  core  elec¬ 
tronics,  and  unmanned  air  vehicles.  Hypothesis 
testing  was  limited  to  those  MDAPs  that  were  in 
or  completed  EMD.  This  final  cleansed  population  of  117  MDAPs 
from  which  valid  samples  were  empirically  analyzed  included  70  MDAPs 
that  used  system  prototype  demonstrations  before  entering  EMD,  and 
47  programs  that  did  not. 


The  MDAP  data  collected  included  available  initial  program  baseline 
dates  for  systems  engineering  technical  reviews  and  key  decision  points 
along  the  program  acquisition  timeline.  Planned  reviews  were  com¬ 
pared  to  actual  event  dates,  and  a  percentage  deviation  was  calculated 
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to  represent  either  schedule  reduction  or  growth.  Data  validity 
and  reliability  for  factors  and  their  constituents  were  assured 
for  comparative  analysis  of  descriptive  statistics,  correlation, 
and  regression  by  using  percentage  deviation  from  plan.  This 
approach  allowed  for  findings  to  be  explained  by  systems  engineer¬ 
ing  progress  rather  than  biased  by  other  potential  factors  associated 
with  the  uniqueness  of  product  type.  Care  was  taken  to  compare  only 
completed  events  so  as  not  to  skew  the  empirical  analysis  results  with 
projected  accomplishments. 


General  Introduction  to  Findings 

A  primary  assumption  in  determining  which  programs  applied 
system  prototype  demonstrations  prior  to  entering  EMD  was  the  fact 
that  all  CTEs  need  to  have  achieved  TRL  6.  Any  program  that  conducted 
a  TRA  and  identified  CTEs  would  have  shown  evidence  that  at  least 
TRL  6  was  achieved  by  Milestone  B,  therefore  validating  that  a  system- 
level  demonstration  had  occurred;  otherwise,  the  Milestone  Decision 
Authority  would  not  have  been  able  to  certify  compliance  with  Title 
10  U.S.C.  §  2366  (NDAA,  2006).  All  programs  after  the  2006  legisla¬ 
tion  would  meet  this  criteria  with  certainty.  Programs  that  conducted 
TRAs  post-2001,  and  before  the  2006  legislation,  would  also  apply  given 
the  need  to  be  consistent  with  then-existing  DoD  5000.02  policy  (DoD, 
2000)  to  perform  technology  maturity  assessments  through  the  appli¬ 
cation  of  TRLs  and  adherence  to  subsequent  Office  of  the  Secretary  of 
Defense  initial  TRA  deskbook  guidance  published  in  2003  (DoD,  2003). 
MDAPs  with  acquisition  strategy  that  included  either  a  Demonstration 
and  Validation  phase  or  Technology  Demonstration  (TD)  phase  were 
also  counted.  These  would  correspond  to  MDAPs  that  held  a  Milestone 
A  event  (or  analogous  Milestone  I  event).  Also  included  were  those  older 
MDAPs  that  employed  Fly-Before-Buy  or  acknowledged  system-level 
demonstrations  that  were  still  part  of  the  active  DoD  portfolio  in  2002, 
and  therefore  were  reported  by  GAO  and  within  the  relevant  data  col¬ 
lection  window  of  this  research  data  population. 

MDAPs  that  were  counted  as  not  using  pre-EMD  system  prototype 
demonstrations  were  those  that  were  initiated  at  or  post-system  devel¬ 
opment  start  (i.e.,  Milestone  B  or  analogous  Milestone  II  event).  MDAPs 
that  entered  the  DAMS  at  production  (i.e.,  Milestone  C  or  analogous 
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Milestone  IIIA  event)  were  not  counted  since  the  acquisition  strategy 
likely  did  not  include  development  activity,  and  therefore  only  accepted 
fully  mature  technologies  into  production. 


Results  and  Findings 

Linear  Relationships  Between  Key  Factor  Constituents 

To  assess  the  strength  and  direction  of  any  linear  relationships,  a 
Pearson  correlation  analysis  was  completed  for  research  factor  constitu¬ 
ents  associated  with  MDAPs  using  system  prototype  demonstrations  to 
assess  the  strength  and  direction  of  any  linear  relationships.  The  impact 
that  system  prototype  demonstrations  have  on  technology  maturity  (e.g., 
TD  span  and  technology  readiness)  was  examined  for  any  relationships 
with  design  maturity  (e.g.,  percent  drawings  released  by  Critical  Design 
Review  [CDR]  and  percent  schedule  change  to  CDR)  and  program  per¬ 
formance  (e.g.,  cost  and  schedule  growth). 

The  Pearson  coefficient  is  based  on  the  method  of  covariance  and  ranges 
from  +1  to  -1,  where  a  value  equivalent  to  zero  (0)  indicates  no  correlation 
between  variables.  As  shown  by  the  sign  of  the  coefficient,  the  direction 
of  the  linear  fit  represents  a  positive  or  negative  relationship  (Laerd 
Statistics,  2013).  Table  1  summarizes  constituent  relationships  for 
MDAPs  that  used  system  prototype  demonstrations  prior  to  EMD.  All 
constituent  pairs  shown  in  Table  1  met  a  0.10  or  higher  level  of  signifi¬ 
cance  (i.e.,  establishing  that  a  relationship  exists). 

Four  constituent  pairs  (AB2,  AB3,  AB4,  and  AB5)  indicated  a  high  degree 
of  association  (i.e.,  strong  correlation)  and  are  characterized  as  follows: 
(a)  any  change  in  the  number  of  CTEs  taken  into  system  development 
will  realize  a  corresponding  change  in  the  time  required  for  TD;  and  (b) 
any  change  in  the  duration  of  time  required  for  TD  will  have  a  similar 
schedule  impact  to  system  development  (i.e.,  EMD  phase),  as  well  as 
an  opposite  impact  on  percent  acquisition  cost  growth.  Therefore,  the 
greater  the  number  of  immature  CTEs  necessary  to  meet  a  capability 
gap,  the  longer  the  TD  phase  will  be  to  reduce  technology  risk  prior  to 
entering  system  development.  Additionally,  given  the  increased  leverage 
of  enhancing  emergent  technologies,  the  EMD  phase  will  likely  be  longer 
to  accommodate  additional  systems  integration  and  test.  The  extended 
TD  phase  would,  with  other  factors  not  considered,  contribute  to  a  reduc¬ 
tion  in  acquisition  cost  growth.  Additionally,  two  constituent  pairs  (AB1 
and  AB6)  were  identified  as  having  a  moderate  degree  of  association  and 
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TABLE  1.  PEARSON  PRODUCT-MOMENT  CORRELATION  ANALYSIS  OF  KEY  CONSTITUENTS 
(MDAPs  with  System  Prototype  Demonstrations  Prior  to  EMD) 
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are  interpreted  as  follows:  (a)  with  a  change  to  EMD  span  time,  there 
is  a  corresponding  opposite  change  in  acquisition  cost  growth  relative 
to  First  Full  Estimates;  and  (b)  with  a  change  in  TD  span  time,  there 
is  a  corresponding  opposite  change  in  acquisition  cycle  time  growth. 
Therefore,  with  longer  TD  spans  to  accommodate  increased  risk  miti¬ 
gation  and  maturation  activities  due  to  increased  number  of  CTEs,  the 
overall  acquisition  cycle  time  can  be  reduced  as  a  result.  Similarly,  with 
longer  EMD  span  times  likely  to  mitigate  complexities  associated  with 
standard  engineering  development  and  complex  integration,  the  percent¬ 
age  of  acquisition  cost  growth  can  be  reduced.  Due  to  direct  relationships 
among  key  constituent  pairs,  the  Pearson  correlation  analysis  indicates 
that  high  potential  exists  for  a  positive  effect  on  program  performance 
when  implementing  effective  risk  reduction  through  the  use  of  system 
prototype  demonstrations. 

System  Prototype  Demonstrations  Provide  a  Positive 
Return-on-lnvestment 

With  the  exception  of  percentage  acquisition  cost  growth  since  the 
First  Full  Estimates  and  percentage  cycle  time  growth  from  program 
start  to  IOC,  Figure  4  shows  that  the  remaining  program  performance 
factor  constituents  show  a  modest  improvement  when  employing  system 
prototype  demonstrations  before  entering  system  development.  MDAPs 
that  leveraged  system  prototype  demonstrations  prior  to  EMD  realized 
a  mean  reduction  in  acquisition  cost  growth  (2006  to  2011)  by  as  much 
as  125%  over  those  that  did  not,  i.e.,  [(17.58-7.82)/7.82]  •  100  =  125%. 
Although  percentage  cycle  time  growth  was  relatively  equal,  with  the 
addition  of  a  TD  phase  (i.e.,  system  prototype  demonstrations),  the  net 
cycle  time  to  the  warfighter  from  both  program  start  and  EMD  start  to 
IOC  was  reduced  by  17%  and  21%,  respectively,  relative  to  MDAPs  that 
did  not  use  system  prototype  demonstrations.  The  average  TD  phase 
span  for  a  sample  of  41  MDAPs  equated  to  3.18  years.  The  noted  improve¬ 
ment  in  percentage  acquisition  cost  growth  measured  from  2006  to 
2011,  as  compared  to  no  improvement  when  measured  against  First  Full 
Estimates  (through  2011),  coincides  with  the  2006  Public  Law  (NDAA, 
2006)  decree  that  all  immature  critical  technologies  are  required  to  be 
demonstrated  in  a  relevant  environment  (i.e.,  TRL  6)  prior  to  receiving 
approval  to  enter  EMD. 

Although  the  empirical  analysis,  as  depicted  in  Figure  4,  shows  a  mini¬ 
mal  difference  in  percentage  cycle  time  growth  from  program  start 
to  IOC  for  those  MDAPs  that  did  and  did  not  use  system  prototype 
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FIGURE  4.  COMPARISON  OF  MDAP 
PERFORMANCE  KEY  CONSTITUENTS 
(With  &  Without  System  Prototype  Demonstrations  Prior  to  EMD) 


Program  Performance  Constituents 

(Normalized  Population) 


%  Acquisition  Cycle  Time  Growth 
(MDAP  Start  to  IOC) 

EMD  Start  to  IOC  (years) 

MDAP  Start  to  IOC  (years) 

EMD  Span  (years) 

%  Acquisition  Cost  Growth  (2006-2011) 

%  Acquisition  Cost  Growth,  Since  1st  Full  Estimate 
(Through  2011) 


■  Non-System  Prototype  MDAPs  ■  System  Prototype  MDAPs 


demonstrations  before  EMD,  the  development  cycle  time  required  to 
IOC  or  from  program  start  to  IOC  is  on  the  average  1.9  years  shorter  for 
MDAPs  using  prototypes.  Coincidentally,  programs  that  used  system 
prototype  demonstrations  had  a  9.8%  lower  mean  total  acquisition  cost 
growth  when  assessed  using  2006  to  2011  data. 

When  comparing  available  MDAP  performance  data  that  are  coincident 
with  the  implementation  of  key  DoD  policy  and  congressional  legislation, 
the  benefits  gained  from  pre-EMD  system  prototype  demonstrations  are 
amplified.  Since  policy  was  introduced  by  DoD  in  2001  to  adopt  TRLs  and 
implement  a  TRA-like  process,  a  23%  reduction  in  mean  total  acquisi¬ 
tion  cost  growth,  relative  to  First  Full  Estimates  (through  2011),  has 
been  realized  (i.e.,  26.2%  cost  growth  prior  to  July  2001  versus  3.64% 
cost  growth  post- July  2001).  Subsequently,  with  the  enactment  of  the 
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NDAA  of  2006  establishing  a  TRL  6  certification  requirement  for  all 
immature  technologies  prior  to  entering  EMD,  a  further  reduction  of 
1.63%  is  observed  (i.e.,  3.64%  cost  growth  post- July  2001  versus  2.1% 
cost  growth  post- January  2006).  Data  were  binned  based  on  when  the 
MDAP  EMD  start  date  occurred  relative  to  the  official  instantiation  of 
the  policy  or  legislation. 

System  Prototype  Demonstrations  Increase 
Technology  Maturity 

Technology  maturity  at  Milestone  B  is  a  significant  factor  since  it 
gauges  the  level  of  technology  risk  carried  forward  into  system  develop¬ 
ment.  Post- January  2006,  the  NDAA  of  2006  ensured  that  a  minimum 
acceptable  TRL  would  need  to  be  achieved  before  awarding  a  develop¬ 
ment  contract.  Just  as  important,  but  not  currently  regulated  by  DoD 
or  legislated  by  Congress,  is  whether  there  should  be  a  best  practice  or 
policy  on  the  total  number  of  CTEs  considered  reasonable  for  an  MDAP 
to  adequately  manage  in  system  development.  The  number  of  CTEs  could 
imply  adequacy  of  requirements  and  extent  of  system  design  complexity 
required  to  meet  operational  needs.  The  data  show  that  when  the  cycle 
time  from  EMD  start  to  IOC  increases,  there  is  a  corresponding  increase 
in  the  number  of  CTEs  that  were  carried  into  EMD.  This  fact,  coupled 
with  the  knowledge  that  EMD  span  increases  with  shorter  TD  spans, 
implies  that  the  greater  the  number  of  immature  critical  technologies 


2% 


n  =  485  CTEs 
(112  MDAPs) 


■  CTEs  @  TRL  4 

■  CTEs  @  TRL  5 
CTEs  @  TRL  6 

■  CTEs  @  TRL  >7 
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introduced  into  EMD,  the  greater  the  technology  risk  transferred  to 
system  development,  and  hence  increased  threat  for  increased  cost  and 
schedule  growth  (i.e.,  reduced  buying  power). 

Figure  5  represents  the  total  number  of  CTEs  reported  by  MDAPs  at 
entry  to  system  development  (i.e.,  Milestone  B),  independent  of  whether 
or  not  system  prototype  demonstrations  were  used  prior  to  Milestone-B. 

The  data  show  77.7%  of  the  MDAPs  at  Milestone  B  reported  CTEs  at  TRL 
6  or  greater  (47.3%  at  TRL  6  and  30.4%  at  >  TRL  7).  The  remaining  22.4% 
of  the  MDAPs  entered  system  development  with  CTEs  less  than  TRL 
6.  Up  until  January  2006,  DoD  was  receptive  to  accepting  and  manag¬ 
ing  technology  risk  in  EMD  based  on  the  establishment  of  a  timely  and 
viable  risk  management  plan.  The  25  MDAPs  that  did  not  meet  minimum 
technology  maturity  requirements  before  entering  system  development 
held  Milestone  B  prior  to  TRL  6  becoming  statutory  law  in  2006  (NDAA, 
2006).  The  mean  number  of  CTEs  entering  system  development  is  four 
for  both  system  prototype  and  non-system  prototype  demonstration 
programs.  MDAPs  using  system  prototype  demonstrations  have  shown 
a  12%  reduction  in  the  number  of  programs  entering  EMD  with  three  to 
five  CTEs.  On  the  other  hand,  the  data  also  show  a  4.1%  increase  in  the 
willingness  of  MDAPs  using  early  system  prototypes  to  carry  6  to  10 
CTEs  into  EMD,  and  correspondingly  a  2.2%  increase  for  those  carrying 
greater  than  10  CTEs. 

System  Prototype  Demonstrations  Increase  Systems 
Design  Maturity 

A  measure  of  design  maturity  is  the  percentage  of  engineering  draw¬ 
ings  available  to  be  released  to  manufacturing  at  both  CDR  and  by 
the  Milestone  C  production  decision  point.  For  MDAPs  sampled  ( n  = 
50),  independent  as  to  whether  system  prototype  demonstrations  were 
employed  prior  to  EMD,  only  48%  of  the  MDAPs  met  DoD  best  practice 
goals  (DoD,  n.d;  DoD,  2011)  of  75%  to  90%  engineering  drawings  complete 
and  releasable  to  manufacturing  by  CDR.  Correspondingly,  only  34%  of 
MDAPs  met  the  GAO  best  practice  goal  (GAO,  2013)  of  at  least  90%  by 
CDR.  The  mean  percentage  engineering  drawings  released  to  manufac¬ 
turing  by  CDR  for  MDAPs  that  used  system  prototype  demonstrations 
prior  to  EMD  is  significantly  greater  than  those  that  did  not  (i.e.,  73.7% 
for  MDAPs  using  prototypes  versus  51.25%  for  MDAPs  not  using  pro¬ 
totypes).  Although  for  CDR  there  is  a  notable  22.5%  improvement  in 
completion  of  engineering  drawings  for  MDAPs  using  system  prototype 
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demonstrations  prior  to  EMD,  this  mark  remains  slightly  short  of  the 
DoD  best  practice  goal  and  16.3%  short  of  GAO's  knowledge  point  best 
practice  goal.  The  mean  percentage  schedule  change  to  CDR  (plan  versus 
actual)  for  those  MDAPs  that  conducted  system  prototype  demonstra¬ 
tions  prior  to  EMD  is  1.84%— significantly  less  than  the  12.45%  realized 
for  those  programs  that  did  not. 


Conclusions 

The  following  quote  (Farrell,  2011)  appropriately  characterizes 
today's  environment  and  the  need  to  apply  systems  engineering  tools 
smartly,  such  as  system  prototype  demonstrations,  to  achieve  early  and 
effective  risk  reduction: 

"Gentleman,  we  have  run  out  of  money.  Now  we  have  to  think.” 
—Sir  Winston  Churchill 

With  the  harsh  realities  of  today's  economics  and  the  need  to  implement 
true  risk  reduction  activities  through  sound  systems  engineering  prac¬ 
tice,  DoD  is  looking  to  leverage  the  knowledge  gained  through  system 
prototype  demonstrations  to  reduce  technical  risk  and  provide  state- 
of-the-art  weapon  systems  to  the  warfighter  sooner— and  at  a  decidedly 
reduced  acquisition  cost. 

The  application  of  system  prototype  demonstrations  to  improve  tech¬ 
nology  maturity  and  accelerate  design  maturity,  as  evidenced  by  the 
findings  of  this  study,  do  indeed  have  a  profound  positive  influence  on  the 


The  knowledge  gained  by  this  study  can  help  the 
government,  in  collaboration  with  industry, 
formulate  more  effective  risk  mitigation 
strategy  for  the  transition  of  influential  enabling 
technologies  into  system  development  such  that 
overall  cycle  time  to  the  warfighter  can  be  reduced . 
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outcome  of  weapon  systems  development  performance.  Data  have  also 
shown  that  with  the  implementation  of  key  policy  and  legislation  to  rein¬ 
force  the  need  to  perform  system-level  prototype  demonstrations  prior 
to  entering  system  development,  MDAP  total  acquisition  cost  growth 
can  be  further  reduced.  Some  key  findings  follow: 

•  The  greater  the  number  of  CTEs  entering  system  development 
(i.e.,  EMD),  the  longer  it  will  take  to  complete  the  preceding 
TD  phase.  Therefore,  it  can  also  be  interpreted  that  the  more 
mature  the  technology  solution  to  fill  a  capability  gap  (i.e.,  lever¬ 
age  of  proven  technology),  the  less  the  dependence  on  TD  and  the 
shorter  the  cycle-time  to  deliver  IOC  to  the  warfighter. 

•  Increased  focus  and  time  invested  during  TD  to  maturate  tech¬ 
nology  solutions  and  reduce  system  development  risk  will  have 
a  positive  contribution  to  reducing  both  acquisition  cost  growth 
and  overall  product  cycle  time  to  the  warfighter. 

•  Although  all  MDAP  CTEs  in  EMD  achieved  at  least  TRL  6  by 
Milestone  B  since  2006,  the  average  number  of  CTEs  carried  into 
EMD  remained  unchanged.  Assuming  the  MDAP  is  not  a  produc¬ 
tion  entry  (i.e.,  Milestone  C)  or  rapid  deployment  acquisition, 
researchers  found  no  evidence  to  suggest  any  policy  or  directives 
that  would  minimize  the  actual  number  of  CTEs  acceptable  for 
entry  into  EMD. 

•  The  average  percent  of  manufacturing  quality  engineering  draw¬ 
ings  available  by  CDR  is  22%  higher  for  MDAPs  that  used  system 
prototype  demonstrations  prior  to  EMD.  There  was  insufficient 
evidence  to  link  the  percentage  of  engineering  drawings  com¬ 
pleted  to  the  amount  of  CTEs  entering  EMD. 

•  MDAPs  with  system  prototype  demonstrations  that  exercised  a 
TD  phase  realized  reduced  product  cycle  time  of  17%  (1.88  years) 
from  program  start  to  IOC,  and  21%  (1.87  years)  for  EMD  start 
to  IOC.  Based  on  a  sampling  of  41  MDAPs,  the  average  span  time 
for  a  TD  phase  has  been  3.18  years. 

The  knowledge  gained  by  this  study  can  help  the  government,  in  collabo¬ 
ration  with  industry,  formulate  more  effective  risk-mitigation  strategy 
for  the  transition  of  influential  enabling  technologies  into  system  devel¬ 
opment  such  that  overall  cycle  time  to  the  warfighter  can  be  reduced. 
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